:W&Bá¾AîO`M cvpqac9wª; 9 /0.29.0+7872fd9d9f86ca5c50617e19cb168122b7f959480.70.0[ÀFù‚" 9ywh7evk04ib cvpqac9wŠÕ cvpqac9wjuju11-uc-san-diegoSPAtop"™ _wandb‚{}  batch_size‚256  dataset_limit‚0.2  dropout‚0.18189335719635055 epochs‚10  hidden_dim‚64 ' learning_rate‚0.0009188811022009657  num_attention_heads‚8  num_branch_encoder_layers‚7  num_embedding_layers‚14  num_encoder_layers‚6Bjjtttt_blocks_cvpqac9wbs2s2ev0zj*jj-spatop-sweepagent-tttt-pairwise-2-dvk5pŠ ¹ÒêÔ€®˜bš3 (H¨¸hp"3.12.10*0.29.0j linux-x86_64± Å›7 cvpqac9w&  _runtime‚0  _wandb.runtime‚0V’ÉjÒä -Linux-6.8.0-138-generic-x86_64-with-glibc2.36CPython 3.12.10 ¸ÒêÔÈïÞ¢*-of*=/data/spatop/options_files/sweep_tttt_jj/config_cvpqac9w.json*-l* /scratch/logs*-n*jjtttt_blocks_cvpqac9w2-m spanet.trainRjacksonjulian2014@gmail.comZ /scratch/logsb*jj-spatop-sweepagent-tttt-pairwise-2-dvk5pr/opt/conda/bin/pythonº df9q2puk92qxzbogcezgbfhg53eufqwq^A´RÉÒÀ€ˆ€’NVIDIA A100-SXM4-80GB˜¢ /€€€¥È.€à’ÑÝ$ª€ äÓ½ÂS NVIDIA A100-SXM4-80GB€€€€À€6"Ampere*(GPU-8cb16037-0025-0eda-9425-82f6843a99c3Ê13.2º df9q2puk92qxzbogcezgbfhg53eufqwqµª½Ç2  requirements.txtW´<W cvpqac9wZF (H¨¸ (H¨¸ (hpð"3.12.10*0.29.0j linux-x86_64¶ºÞ‰Y cvpqac9wZH (H¨¸ (H¨¸ (8hpð"3.12.10*0.29.0j linux-x86_64Mý*  cvpqac9wb trainer/global_step20Ö¦=\  cvpqac9wZK (H¨¸ (H¨¸(8hpð"3.12.10*0.29.0j linux-x86_64%'g/  cvpqac9wb*"trainer/global_step2ÙûMK  cvpqac9wj: ¹ÒêÔБõ±*LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] oÓ“  cvpqac9wj  ºÒêÔÀûŒ‹î â”â”â”â”┳â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”┓ ┃ ┃ Name ┃ Type ┃ Params ┃ Mode ┃ FLOPs ┃ ┡â”â”â”╇â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”┩ │ 0 │ embedding │ MultiInputVectorEmbed… │ 220 K │ train │ 0 │ │ 1 │ encoder │ JetEncoderWithPairwise │ 498 K │ train │ 0 │ │ 2 │ branch_decoders │ ModuleList │ 13.6 M │ train │ 0 │ │ 3 │ regression_decoder │ RegressionDecoder │ 0 │ train │ 0 │ │ 4 │ classification_decoder │ ClassificationDecoder │ 0 │ train │ 0 │ │ │ other params │ n/a │ 48 │ n/a │ n/a │ └───┴────────────────────────┴────────────────────────┴────────┴───────┴───────┘ ̾†‘\ cvpqac9wjÊ ºÒêÔزòŒ·Trainable params: 14.3 M Non-trainable params: 86 Total params: 14.3 M Total estimated model params size (MB): 57.161 Modules in train mode: 5928 Modules in eval mode: 0 Total FLOPs: 0 ꄱëq cvpqac9wj` ºÒêԸˆŽPwandb: WARNING Config item 'hidden_dim' was locked by 'sweep' (ignored update). š‘m{ cvpqac9wjj ºÒêÔ㺎Zwandb: WARNING Config item 'num_embedding_layers' was locked by 'sweep' (ignored update). æ Ñgy cvpqac9wjh ºÒêÔ€ àŽXwandb: WARNING Config item 'num_encoder_layers' was locked by 'sweep' (ignored update). ˆY„u€ cvpqac9wjo ºÒêÔ¨ÈêŽ_wandb: WARNING Config item 'num_branch_encoder_layers' was locked by 'sweep' (ignored update). Çv=z cvpqac9wji ºÒêÔèôðŽYwandb: WARNING Config item 'num_attention_heads' was locked by 'sweep' (ignored update). ö±†Ot cvpqac9wjc ºÒêÔ¨¡÷ŽSwandb: WARNING Config item 'dataset_limit' was locked by 'sweep' (ignored update). ⎹q cvpqac9wj` ºÒêÔ€ÉüŽPwandb: WARNING Config item 'batch_size' was locked by 'sweep' (ignored update). ŽFD-t cvpqac9wjc ºÒêÔ…ƒSwandb: WARNING Config item 'learning_rate' was locked by 'sweep' (ignored update). Í’#[n cvpqac9wj] ºÒêÔÀãˆMwandb: WARNING Config item 'dropout' was locked by 'sweep' (ignored update). ×Jˆ m cvpqac9wj\ ºÒêÔäLwandb: WARNING Config item 'epochs' was locked by 'sweep' (ignored update). 0Ö~0Ù cvpqac9w*Ç  transformer_dim‚64  transformer_dim_scale‚2.0  initial_embedding_dim‚64  position_embedding_dim‚64 ! num_branch_embedding_layers‚3  num_jet_embedding_layers‚0  num_jet_encoder_layers‚2  num_detector_layers‚2  num_regression_layers‚3  num_classification_layers‚3  split_symmetric_attention‚1 ! transformer_activation‚"gelu"  skip_connections‚1 ( "initial_embedding_skip_connections‚1  linear_block_type‚"GRU"  transformer_type‚"Gated"  linear_activation‚"gelu"  normalization‚ "LayerNorm"  masking‚ "Filling" linear_prelu_activation‚true  use_pairwise_interactions‚1  num_pairwise_features‚4  pairwise_embedding_dim‚8  pairwise_input_source‚"all"  pairwise_block_embeddings‚1  pairwise_cross_type‚true  pairwise_feature_set‚""  pairwise_eq2to2‚false  use_moe‚false  num_experts‚8  num_experts_per_tok‚2  moe_loss_scale‚0.01 F event_info_file‚2"/data/spatop/event_files/tttt/tttt_hadronic.yaml" = training_file‚+"/data/spatop/tttt_15M/h5/tttt_training.h5"  validation_file‚""  testing_file‚""  normalize_features‚1  limit_to_num_jets‚0  balance_particles‚0  balance_jets‚0 ! balance_classifications‚false  partial_events‚1  dataset_randomization‚0  train_validation_split‚0.95  limit_val_batches‚1.0  num_dataloader_workers‚0  mask_sequence_vectors‚1  combine_pair_loss‚"min"  optimizer‚"AdamW"  focal_gamma‚0.0  combinatorial_scale‚0.0 # learning_rate_warmup_epochs‚1.0  learning_rate_cycles‚1  assignment_loss_scale‚1.0  detection_loss_scale‚1.0  kl_loss_scale‚0.0  regression_loss_scale‚0.0 ! classification_loss_scale‚0.0 ! use_mass_classification‚false  mass_classes‚[]  balance_losses‚true  l2_penalty‚0.0002  gradient_clip‚10.0 num_gpu‚1  verbose_output‚false  usable_gpus‚""  trial_time‚"" $ trial_output_dir‚"./test_output"˜~ë cvpqac9wjÙ ºÒêÔˆÃÏ¢È/opt/conda/lib/python3.12/site-packages/pytorch_lightning/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead. Š÷ßN cvpqac9wj¼ ºÒêÔÀÿ–£«/opt/conda/lib/python3.12/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:434: The 'val_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=3` in the `DataLoader` to improve performance. —îD¸ë cvpqac9wjÙ ÁÒêÔÂœòÈ/opt/conda/lib/python3.12/site-packages/pytorch_lightning/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead. '#^P cvpqac9wj¾ ÁÒêÔ¨·šó­/opt/conda/lib/python3.12/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:434: The 'train_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=3` in the `DataLoader` to improve performance. »‚(:† ÈÒêÔ›ò²ö gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.945388793945312) gpu.0.memoryAllocatedBytes‚ 7684030464 gpu.0.temp‚30.0 gpu.0.powerWatts‚113.934' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.7868 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~¢m±X‚(:Î ÈÒêԥ܂ disk.nvme0n1.in‚ 107.16796875$ disk.nvme0n1.out‚226.92822265625$ memory_percent‚96.55284881591797 proc.cpu.threads‚17 network.sent‚48338 disk.sda1.in‚0# proc.memory.rssMB‚ 2812.89453125 cpu‚26.60630954547204) disk./.usagePercent‚78.88264431329031 disk.sda1.out‚0 disk.drbd1260.out‚ 0.00390625 disk.sda2.out‚0! disk.dm-0.out‚226.92822265625 disk.sda3.in‚0% proc.memory.availableMB‚ 564.78125* proc.memory.percent‚17.168545722961426 network.recv‚66780$ disk./.usageGB‚1175.408405303955 disk.dm-1.in‚0 disk.dm-1.out‚0.59375 disk.sda2.in‚0 disk.dm-0.in‚ 107.16796875 disk.sda.in‚0 disk.sda.out‚0.59375 disk.drbd1260.in‚0 disk.sda3.out‚0.59375" disk.nvme0c0n1.in‚ 107.16796875& disk.nvme0c0n1.out‚226.92822265625gÌy Ž ‚(:„ ×ÒêÔ•Ý£÷ gpu.0.gpu‚5.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚29.0 gpu.0.powerWatts‚98.962' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.7924 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Š#=˜b!‚(:Ø ×ÒêԮɗú disk.sda1.out‚0 disk.dm-1.in‚0! disk.dm-0.out‚601.63916015625 disk.sda.in‚0 disk.sda3.out‚ 0.94140625$ memory_percent‚96.56798839569092 network.recv‚71175) disk./.usagePercent‚78.89106491310653$ disk./.usageGB‚1175.533878326416& disk.nvme0c0n1.out‚601.63916015625 disk.dm-1.out‚ 0.94140625 disk.drbd1260.out‚ 0.00390625 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda1.in‚0 disk.drbd1260.in‚0 disk.sda2.in‚0( proc.memory.availableMB‚ 562.30078125" proc.memory.rssMB‚ 2813.9609375 proc.cpu.threads‚17 network.sent‚58186 disk.nvme0n1.in‚ 133.12109375 disk.dm-0.in‚ 133.12109375 disk.sda.out‚ 0.94140625( proc.memory.percent‚17.1750545501709 cpu‚26.332667608119056" disk.nvme0c0n1.in‚ 133.12109375$ disk.nvme0n1.out‚601.63916015625\¥a`"‚(:† æÒêÔÓžÆö gpu.0.gpu‚24.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚105.766' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.1532 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Òî÷`#‚(:Ö æÒêÔÿ»‡ù disk.sda3.in‚0 disk.dm-1.out‚ 1.3515625 disk.sda2.out‚0$ memory_percent‚96.56994342803955( proc.memory.availableMB‚ 561.98046875 proc.cpu.threads‚17 network.sent‚65148) disk./.usagePercent‚78.89102907205174 disk.dm-0.in‚ 133.12109375" disk.nvme0c0n1.in‚ 133.12109375 disk.drbd1260.out‚ 0.01171875 cpu‚26.143636982358725! disk.dm-0.out‚799.06005859375 disk.sda.out‚ 1.3515625 disk.sda3.out‚ 1.3515625 disk.nvme0n1.in‚ 133.12109375$ disk.nvme0n1.out‚799.06005859375 disk.dm-1.in‚0) proc.memory.percent‚17.17519760131836 network.recv‚76523% disk./.usageGB‚1175.5333442687988& disk.nvme0c0n1.out‚799.06005859375 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.sda2.in‚0! proc.memory.rssMB‚ 2813.984375 disk.sda.in‚0_檎$‚(:„ õÒêÔÒÝÊö gpu.0.gpu‚5.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚92.744' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.5488 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0xÙN°c%‚(:Ù õÒêÔì“Áú% disk./.usageGB‚1175.6353912353516 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.01171875 disk.sda2.out‚0 disk.dm-0.in‚ 133.12109375 disk.sda3.out‚ 1.83984375 disk.sda2.in‚0! disk.dm-0.out‚905.34521484375 disk.sda3.in‚0& disk.nvme0c0n1.out‚905.34521484375$ disk.nvme0n1.out‚905.34521484375$ memory_percent‚96.57785892486572! proc.memory.rssMB‚ 2813.984375 cpu‚26.00661477516867 disk.dm-1.in‚0 disk.dm-1.out‚ 1.83984375 disk.sda.in‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 560.68359375) proc.memory.percent‚17.17519760131836 network.recv‚106187 disk.sda.out‚ 1.83984375" disk.nvme0c0n1.in‚ 133.12109375 disk.nvme0n1.in‚ 133.12109375 disk.sda1.in‚0 proc.cpu.threads‚17 network.sent‚84386) disk./.usagePercent‚78.89787752960021§ÃW&‚(:† „ÓêÔ¸ÍÞ÷ gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚112.742' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.5484 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0´#žãa'‚(:× „ÓêÔÝÉ—û disk.dm-1.in‚0 disk.drbd1260.in‚0 proc.cpu.threads‚17% disk.nvme0n1.out‚1241.07568359375 disk.sda2.in‚0" disk.dm-0.out‚1241.07568359375 disk.sda.out‚ 2.2734375$ memory_percent‚96.58222198486328% proc.memory.availableMB‚ 559.96875) proc.memory.percent‚17.17522144317627 cpu‚25.906472308761103% disk./.usageGB‚1175.6348762512207' disk.nvme0c0n1.out‚1241.07568359375 disk.drbd1260.out‚0.015625 disk.sda2.out‚0 disk.dm-0.in‚ 133.12109375 network.recv‚111980 disk.sda1.out‚0 disk.dm-1.out‚ 2.2734375 disk.sda.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 2.2734375# proc.memory.rssMB‚ 2813.98828125 network.sent‚91309) disk./.usagePercent‚78.89784296858309" disk.nvme0c0n1.in‚ 133.12109375 disk.nvme0n1.in‚ 133.12109375 disk.sda1.in‚0²t„(‚(:† “ÓêÔ·ŽÂõ gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚117.156' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.4312 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÒäYÀZ)‚(:Ð “ÓêÔÚ‚Â÷$ memory_percent‚96.58451080322266% proc.memory.availableMB‚ 559.59375" disk.dm-0.out‚1491.50537109375 disk.sda.in‚0' disk.nvme0c0n1.out‚1491.50537109375 disk.nvme0n1.in‚ 322.71484375 disk.sda1.in‚0" proc.memory.rssMB‚ 2813.9921875 network.sent‚97307 disk.sda3.in‚0 disk.drbd1260.in‚0) proc.memory.percent‚17.17524528503418 cpu‚25.823947078890825 proc.cpu.threads‚17 network.recv‚115747 disk.sda2.out‚0 disk.dm-0.in‚ 322.71484375" disk.nvme0c0n1.in‚ 322.71484375% disk.nvme0n1.out‚1491.50537109375 disk.sda1.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚2.640625' disk./.usagePercent‚78.916265270747$ disk./.usageGB‚1175.909381866455 disk.sda2.in‚0 disk.sda.out‚2.640625 disk.sda3.out‚2.640625 disk.drbd1260.out‚0.015625߯ëÝ*‚(:† ¢ÓêÔ¡ÂÈú gpu.0.gpu‚18.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚101.848' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.3696 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ý!žÈe+‚(:Û ¢ÓêÔ©‰ðú disk.sda.out‚ 3.1484375# memory_percent‚96.5869665145874) proc.memory.percent‚17.17534065246582 proc.cpu.threads‚17 disk.drbd1260.in‚0 disk.sda2.in‚0 network.sent‚102265% disk./.usageGB‚1175.9083442687988 disk.nvme0n1.in‚ 322.71484375% disk.nvme0n1.out‚1981.15380859375 disk.sda3.in‚0 disk.drbd1260.out‚ 0.01953125 disk.sda2.out‚0" disk.dm-0.out‚1981.15380859375" disk.nvme0c0n1.in‚ 322.71484375' disk.nvme0c0n1.out‚1981.15380859375( proc.memory.availableMB‚ 559.19140625 network.recv‚119906) disk./.usagePercent‚78.91619563669771 disk.sda1.out‚0 disk.dm-1.out‚ 3.1484375 disk.sda.in‚0 disk.sda3.out‚ 3.1484375" proc.memory.rssMB‚ 2814.0078125 cpu‚25.759671724847486 disk.sda1.in‚0 disk.dm-1.in‚0 disk.dm-0.in‚ 322.71484375=¹.53,b/ lr-AdamW/pg1"trainer/global_step2:JP–S™Ü3-b/ lr-AdamW/pg2"trainer/global_step2:JP—M¼%/.+ _wandbruntime‚108  _runtime‚108¾—<’À/» & lr-AdamW/pg1‚3.724166584602756e-05 & lr-AdamW/pg2‚3.724166584602756e-05 trainer/global_step‚49 $ _timestamp‚1.788520869877656e+09 _runtime‚ 108.099173062 _step‚0I¤AÂ0½ & lr-AdamW/pg1‚3.724166584602756e-05 & lr-AdamW/pg2‚3.724166584602756e-05 trainer/global_step‚49 $ _timestamp‚1.788520869877656e+09 _runtime‚ 108.099173062 _step‚0N~¨t1bp MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed"trainer/global_step2:JPgEa2b] :DeviceStatsMonitor.on_train_batch_start/num_oom_rejections"trainer/global_step2:JP‡)h3bd ADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak"trainer/global_step2:JP©ºtDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak"trainer/global_step2:JP6»e8ba >DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed"trainer/global_step2:JP»*ljn9bj GDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current"trainer/global_step2:JP“Çwär:bn KDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated"trainer/global_step2:JP ¶ãVs;bo LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak"trainer/global_step2:JP<-…jbd ADeviceStatsMonitor.on_train_batch_start/active.small_pool.current"trainer/global_step2:JPM’ŠËn?bj GDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed"trainer/global_step2:JPMv¦k@bg DDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated"trainer/global_step2:JPª[]AbY 6DeviceStatsMonitor.on_train_batch_start/max_split_size"trainer/global_step2:JPoŒísBbo LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated"trainer/global_step2:JPµ›‘ÝqCbm JDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current"trainer/global_step2:JP–ì!.rDbn KDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated"trainer/global_step2:JPÑ92>nEbj GDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed"trainer/global_step2:JP¶ÍÄÉlFbh EDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak"trainer/global_step2:JPñ]à™vGbr ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current"trainer/global_step2:JPZ'nHbj GDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak"trainer/global_step2:JP¤ýÑfIbb ?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak"trainer/global_step2:JPéýÙcJb_ ³_Mb[ 8DeviceStatsMonitor.on_train_batch_start/num_device_alloc"trainer/global_step2:JPºlÉÝfNbb ?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed"trainer/global_step2:JPaùr-gObc @DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed"trainer/global_step2:JPCÁ²î^PbZ 7DeviceStatsMonitor.on_train_batch_start/active.all.peak"trainer/global_step2:JPL­r¦eQba >DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak"trainer/global_step2:JP®£e1sRbo LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated"trainer/global_step2:JPmÀÖmSbi FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed"trainer/global_step2:JP+…–eTba >DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak"trainer/global_step2:JPy†£dUb` =DeviceStatsMonitor.on_train_batch_start/segment.all.allocated"trainer/global_step2:JPi&Ø%hVbd ADeviceStatsMonitor.on_train_batch_start/active.large_pool.current"trainer/global_step2:JP€|lWbh EDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current"trainer/global_step2:JPÄÕó>tXbp MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed"trainer/global_step2:JP¬r{vlYbh EDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated"trainer/global_step2:JPŽ¦¹+iZbe BDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak"trainer/global_step2:JP¼çĸh[bd ADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed"trainer/global_step2:JPÀ×q]e\ba >DeviceStatsMonitor.on_train_batch_start/allocation.all.current"trainer/global_step2:JP©ŠÌm]bi FDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated"trainer/global_step2:JPg—d§o^bk HDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current"trainer/global_step2:JP¶_b_b^ ;DeviceStatsMonitor.on_train_batch_start/segment.all.current"trainer/global_step2:JP}jÇj`bf CDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current"trainer/global_step2:JP¡«µ¾bab^ ;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak"trainer/global_step2:JPB±fbbb ?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak"trainer/global_step2:JPg~¶b^cbZ 7DeviceStatsMonitor.on_train_batch_start/num_device_free"trainer/global_step2:JP磉õkdbg DDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current"trainer/global_step2:JP€·Šâoebk HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed"trainer/global_step2:JP^­[Ùmfbi FDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak"trainer/global_step2:JPUŠççggbc @DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak"trainer/global_step2:JPííïcshbo LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated"trainer/global_step2:JP82çnibj GDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed"trainer/global_step2:JPÔY·åojbk HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed"trainer/global_step2:JPpÔqbakb] :DeviceStatsMonitor.on_train_batch_start/active.all.current"trainer/global_step2:JPv ¢ìnlbj GDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak"trainer/global_step2:JP4t^qimbe BDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak"trainer/global_step2:JP^ ô inbe BDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed"trainer/global_step2:JP\ªÔkobg DDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated"trainer/global_step2:JPÜUþŽppbl IDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current"trainer/global_step2:JPÂõ­Hiqbe BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated"trainer/global_step2:JP·ná_rb[ 8DeviceStatsMonitor.on_train_batch_start/active.all.freed"trainer/global_step2:JP‹­·,nsbj GDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current"trainer/global_step2:JP¸çTÇgtbc @DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak"trainer/global_step2:JPÊODdiube BDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current"trainer/global_step2:JPþÍ´_vb[ 8DeviceStatsMonitor.on_train_batch_start/segment.all.peak"trainer/global_step2:JP~©7àiwbe BDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current"trainer/global_step2:JP…Ô·Õjxbf CDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated"trainer/global_step2:JPà¤qybm JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated"trainer/global_step2:JP-Ò xzbt QDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated"trainer/global_step2:JPSòq{bm JDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current"trainer/global_step2:JPxBm|bi FDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak"trainer/global_step2:JPô°œYg}bc @DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed"trainer/global_step2:JPQïlf~bb ?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak"trainer/global_step2:JPA'Œcb_ t¦bo LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated"trainer/global_step2:JPžêk§bf CDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed"trainer/global_step2:JPÿŽ]o¨bj GDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated"trainer/global_step2:JPÊ’E}w©br ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current"trainer/global_step2:JP•à09qªbl IDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current"trainer/global_step2:JPÃ5Ö€q«bl IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated"trainer/global_step2:JP8]ú0¬+ _wandbruntime‚108  _runtime‚108^±©Ç­O WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 1982035655680 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30749 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30506 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 1982053658624 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 2200444053504 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 ]MDeviceStatsMonitor.on_train_batch_staØX0À#rt/inactive_split_bytes.small_pool.freed‚ 99146089984 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 1987220283232 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 2079179365888 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 302743552 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30749 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1284 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 97143710208 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1265 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051901 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 1890123852400 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚49 _step‚1 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 1982035655680 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564257792 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30506 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 2079430270464 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚2252739 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966132736 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚1351346 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 2200746797056 aQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 99424893440 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 2079179365888 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚350646 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 NCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚926241 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚899466 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 97096430832 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚548820 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚349951 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 325796352 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281221 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 2079179365888 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚350646 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 NCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚926241 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚899466 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 97096430832 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚548820 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚349951 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 325796352 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281221 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30506 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 1982053658624 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 2200444053504 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 ]MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 99146089984 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 1987220283232 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 2079179365888 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 302743552 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30749 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1284 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 97143710208 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1265 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051901 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 1890123852400 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚49 _step‚1 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 1982035655680 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564257792 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30506 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 2079430270464 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚2252739 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966132736 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚1351346 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 2200746797056 aQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 99424893440 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 HDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed"trainer/global_step2:JPô\ŽÙbáb] :DeviceStatsMonitor.on_train_batch_end/allocation.all.freed"trainer/global_step2:JPu-Njfâba >DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed"trainer/global_step2:JPŸ§Ù\ãbW 4DeviceStatsMonitor.on_train_batch_end/max_split_size"trainer/global_step2:JPßòþmäbh EDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak"trainer/global_step2:JP ÇÂdåb_ DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated"trainer/global_step2:JPë8{]êbX 5DeviceStatsMonitor.on_train_batch_end/num_device_free"trainer/global_step2:JP0MÛfëba >DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed"trainer/global_step2:JP×SÕýmìbh EDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current"trainer/global_step2:JPü®Ý…kíbf CDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current"trainer/global_step2:JPFEw0sîbn KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed"trainer/global_step2:JPw.ƒ`ïb[ 8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections"trainer/global_step2:JPþ±Lkrðbm JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated"trainer/global_step2:JPk4°ùfñba >DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed"trainer/global_step2:JP©RÀiòbd ADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed"trainer/global_step2:JP¸£Ë´hóbc @DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current"trainer/global_step2:JPOjZoôbj GDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current"trainer/global_step2:JPyè¢hõbc @DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current"trainer/global_step2:JPdïðŸpöbk HDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current"trainer/global_step2:JP©{q÷bl IDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated"trainer/global_step2:JP„3hgøbb ?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak"trainer/global_step2:JPh&vPfùba >DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak"trainer/global_step2:JP_Þhúbc @DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current"trainer/global_step2:JPaΉâVûbQ .DeviceStatsMonitor.on_train_batch_end/num_ooms"trainer/global_step2:JPØ8Ömübh EDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated"trainer/global_step2:JPz×2nýbi FDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current"trainer/global_step2:JPÙÖlþbg DDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated"trainer/global_step2:JPfŠZØgÿbb ?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed"trainer/global_step2:JP‰Ì‡=d€b_ DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak"trainer/global_step2:JPè0Xeb` =DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed"trainer/global_step2:JP»í’]žbX 5DeviceStatsMonitor.on_train_batch_end/active.all.peak"trainer/global_step2:JPVyÍ…lŸbg DDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed"trainer/global_step2:JP·PÀÙf ba >DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current"trainer/global_step2:JPîq°c¡b^ ;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak"trainer/global_step2:JP|Ðla¢b\ 9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak"trainer/global_step2:JPFÝÀj£be BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated"trainer/global_step2:JP9ê1r¤bm JDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak"trainer/global_step2:JPÔd:´d¥b_ DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _runtime‚ 110.492983917 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚1378064 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306924544 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 2026463116746 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 2244120665088 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚917317 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 SCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 99107736064 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 2120429428736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1284 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282984448 trainer/global_step‚49 WGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 99340651008 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 325796352 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 101138731520 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚356998 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 2143005873664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 1927654209824 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 1927637170464 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 PEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚944516 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 2244427589632 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚559620 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 LADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚944516 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 2120680346624 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 2021339695616 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 2021321692672 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚2297706 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚916618 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚2322580 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 2142981933568 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZJDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 99054920026 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚1353192 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚699 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309#XPB—&¬‘M ZJDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 99340651008 NDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 2244120665088 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚917317 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 SCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 99107736064 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 2120429428736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1284 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282984448 trainer/global_step‚49 WGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 99340651008 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 325796352 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 101138731520 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚356998 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 2143005873664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 1927654209824 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 1927637170464 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 PEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚944516 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 2244427589632 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚559620 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 LADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚944516 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 2120680346624 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 2021339695616 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 2021321692672 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚2297706 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚916618 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚2322580 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 2142981933568 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZJDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 99054920026 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚1353192 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚699 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚357695 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _runtime‚ 110.492983917 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚1378064 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306924544 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 2026463116746 E loss/SRqqt4/assignment_loss"trainer/global_step2:JPûÿMåA°b< loss/FBt1/assignment_loss"trainer/global_step2:JPÍçk¹A±b< loss/FBt3/assignment_loss"trainer/global_step2:JPˆFâB²b= loss/SRqqt1/detection_loss"trainer/global_step2:JPçD]@³b; loss/FBt4/detection_loss"trainer/global_step2:JPš7lMA´b< loss/FBt4/assignment_loss"trainer/global_step2:JPU`òBµb= loss/SRqqt3/detection_loss"trainer/global_step2:JPâ8y@¶b; loss/FBt2/detection_loss"trainer/global_step2:JP-}’Þ@·b; loss/FBt3/detection_loss"trainer/global_step2:JPuš:A¸b< loss/FRt2/assignment_loss"trainer/global_step2:JPowì”@¹b; loss/FRt2/detection_loss"trainer/global_step2:JPðŸCëCºb> loss/SRqqt1/assignment_loss"trainer/global_step2:JPL¹qC»b> loss/SRqqt3/assignment_loss"trainer/global_step2:JP‘Ÿ»Ò@¼b; loss/FRt3/detection_loss"trainer/global_step2:JP*­ÉG@½b; loss/FRt1/detection_loss"trainer/global_step2:JPŸ¶@C¾b> loss/SRqqt2/assignment_loss"trainer/global_step2:JPŠ Â¥A¿b< loss/FRt3/assignment_loss"trainer/global_step2:JPx‚ßAÀb< loss/FBt2/assignment_loss"trainer/global_step2:JPŸ)BÁb= loss/SRqqt2/detection_loss"trainer/global_step2:JPD!­¿7Âb2 loss/total_loss"trainer/global_step2:JPáXŸ0-Ãb( epoch"trainer/global_step2:JPÞ(û AÄb< loss/FRt4/assignment_loss"trainer/global_step2:JP€RN@@Åb; loss/FRt4/detection_loss"trainer/global_step2:JPG_n¢BÆb= loss/SRqqt4/detection_loss"trainer/global_step2:JPz·}0Ç+ _wandbruntime‚110  _runtime‚110yž.TGÈÁ /loss/FRt1/assignment_loss‚7.117890619462536 0loss/SRqqt4/assignment_loss‚5.47494617055674 /loss/FBt1/assignment_loss‚2.795114820653742 0loss/FBt3/assignment_loss‚2.7436395552423267 1loss/SRqqt1/detection_loss‚3.4324973424275718 .loss/FBt4/detection_loss‚9.450235920517068 trainer/global_step‚49 _step‚3 0loss/FBt4/assignment_loss‚2.7142014127028617 /loss/SRqqt3/detection_loss‚6.65256028679701 .loss/FBt2/detection_loss‚4.395468653012545 .loss/FBt3/detection_loss‚4.793654030085438 /loss/FRt2/assignment_loss‚7.211894008091518 .loss/FRt2/detection_loss‚4.807302966288158 % _timestamp‚1.7885208722772079e+09 _runtime‚ 110.500881532 1loss/SRqqt1/assignment_loss‚5.366911788781484 1loss/SRqqt3/assignment_loss‚5.495133876800537 /loss/FRt3/detection_loss‚6.3625831099656915 .loss/FRt1/detection_loss‚2.383071735795144 1loss/SRqqt2/assignment_loss‚5.390331069035317 /loss/FRt3/assignment_loss‚7.238344761041494 0loss/FBt2/assignment_loss‚2.7370841931074095 0loss/SRqqt2/detection_loss‚2.529182122936889 &loss/total_loss‚127.05397617401188 epoch‚0 /loss/FRt4/assignment_loss‚7.352667938579213 /loss/FRt4/detection_loss‚2.4088049472281425 0loss/SRqqt4/detection_loss‚2.650908397235831 .loss/FBt1/detection_loss‚15.54954644766721};M–KÉÅ /loss/FRt2/assignment_loss‚7.211894008091518 .loss/FRt2/detection_loss‚4.807302966288158 % _timestamp‚1.7885208722772079e+09 _runtime‚ 110.500881532 1loss/SRqqt1/assignment_loss‚5.366911788781484 1loss/SRqqt3/assignment_loss‚5.495133876800537 /loss/FRt3/detection_loss‚6.3625831099656915 .loss/FRt1/detection_loss‚2.383071735795144 1loss/SRqqt2/assignment_loss‚5.390331069035317 /loss/FRt3/assignment_loss‚7.238344761041494 0loss/FBt2/assignment_loss‚2.7370841931074095 0loss/SRqqt2/detection_loss‚2.529182122936889 &loss/total_loss‚127.05397617401188 epoch‚0 /loss/FRt4/assignment_loss‚7.352667938579213 /loss/FRt4/detection_loss‚2.4088049472281425 0loss/SRqqt4/detection_loss‚2.650908397235831 .loss/FBt1/detection_loss‚15.54954644766721 /loss/FRt1/assignment_loss‚7.117890619462536 0loss/SRqqt4/assignment_loss‚5.47494617055674 /loss/FBt1/assignment_loss‚2.795114820653742 0loss/FBt3/assignment_loss‚2.7436395552423267 1loss/SRqqt1/detection_loss‚3.4324973424275718 .loss/FBt4/detection_loss‚9.450235920517068 trainer/global_step‚49 _step‚3 0loss/FBt4/assignment_loss‚2.7142014127028617 /loss/SRqqt3/detection_loss‚6.65256028679701 .loss/FBt2/detection_loss‚4.395468653012545 .loss/FBt3/detection_loss‚4.793654030085438=¢ê,jÊ‚(:ß ±ÓêÔüßÌ÷* proc.memory.percent‚17.177844047546387 proc.cpu.threads‚17" disk.dm-0.out‚2173.74365234375 disk.sda.in‚0 disk.nvme0n1.in‚ 322.71484375 disk.sda1.in‚0$ memory_percent‚96.60162925720215$ disk./.usageGB‚1175.908405303955 disk.sda.out‚ 3.62109375 disk.sda3.in‚0' disk.nvme0c0n1.out‚2173.74365234375% disk.nvme0n1.out‚2173.74365234375 disk.drbd1260.in‚0' proc.memory.availableMB‚ 556.7890625 cpu‚25.705803028051413 network.sent‚114941 network.recv‚127138 disk.sda3.out‚ 3.62109375" disk.nvme0c0n1.in‚ 322.71484375 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.01953125 disk.sda2.in‚0 disk.sda2.out‚0) disk./.usagePercent‚78.91619973281824 disk.dm-0.in‚ 322.71484375 disk.sda1.out‚0 disk.dm-1.out‚ 3.62109375# proc.memory.rssMB‚ 2814.41796875¼†âœË‚(:‘ ±ÓêÔ°‘Øù gpu.0.gpu‚23.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚104.707' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚20.941399999999998 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÑžœÌ‚(:‘ ÀÓêÔë…Ùö gpu.0.gpu‚29.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚126.786' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.357200000000002 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0øG¹fÍ‚(:Û ÀÓêÔ—“€ø" disk.dm-0.out‚2322.89794921875 disk.sda.in‚0" disk.nvme0c0n1.in‚ 322.71484375' disk.nvme0c0n1.out‚2322.89794921875$ memory_percent‚96.60382270812988' proc.memory.availableMB‚ 556.4296875# proc.memory.rssMB‚ 2814.41796875 network.sent‚165203 network.recv‚133723 disk.dm-1.in‚0 disk.dm-1.out‚ 4.0546875 disk.drbd1260.out‚ 0.0234375 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.dm-0.in‚ 322.71484375 disk.sda.out‚ 4.0546875% disk.nvme0n1.out‚2322.89794921875 disk.sda1.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚78.91620075684838 disk.sda3.out‚ 4.0546875 disk.nvme0n1.in‚ 322.71484375* proc.memory.percent‚17.177844047546387 cpu‚25.65956040550222% disk./.usageGB‚1175.9084205627441‡¿œÎ‚(:‘ ÏÓêÔÖ©Ž÷ gpu.0.gpu‚43.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚128.955' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.791000000000004 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ãí¾k_Ï‚(:Ô ÏÓêÔØŸãø% disk./.usageGB‚1175.9084396362305 disk.dm-0.in‚ 471.640625 disk.nvme0c0n1.in‚ 471.640625 disk.nvme0n1.in‚ 471.640625$ disk.nvme0n1.out‚2757.9619140625 disk.sda1.in‚0 disk.sda1.out‚0 network.sent‚170832 network.recv‚137751 disk.sda2.in‚0! disk.dm-0.out‚2757.9619140625 disk.sda3.in‚0 disk.dm-1.in‚0 disk.drbd1260.in‚0$ memory_percent‚96.60437107086182 disk.sda3.out‚ 4.45703125 disk.dm-1.out‚ 4.45703125( proc.memory.availableMB‚ 556.33984375! proc.memory.rssMB‚ 2814.421875& disk.nvme0c0n1.out‚2757.9619140625 disk.sda2.out‚0 disk.sda.in‚0 disk.sda.out‚ 4.45703125 disk.drbd1260.out‚ 0.0234375* proc.memory.percent‚17.177867889404297 cpu‚25.62011998905074 proc.cpu.threads‚17) disk./.usagePercent‚78.91620203688605|b}‘Ђ(:† ÞÓêÔ僖ö gpu.0.gpu‚13.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚119.037' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.8074 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õÓûÌHÑ‚(:½ ÞÓêÔÆ­”ø disk.nvme0n1.out‚ 3087.984375 disk.sda1.in‚0 disk.sda.in‚0 disk.sda3.in‚0 disk.sda3.out‚4.96875 cpu‚25.579551918219813 network.sent‚176513) disk./.usagePercent‚78.87407830119463" disk.nvme0c0n1.out‚ 3087.984375 disk.dm-1.in‚0 disk.dm-1.out‚4.96875 disk.drbd1260.out‚ 0.02734375 disk.sda2.in‚0 disk.sda2.out‚0 network.recv‚141758% disk./.usageGB‚1175.2807655334473 disk.nvme0c0n1.in‚ 471.640625 disk.nvme0n1.in‚ 471.640625 disk.sda1.out‚0 disk.dm-0.in‚ 471.640625 disk.dm-0.out‚ 3087.984375 disk.sda.out‚4.96875 disk.drbd1260.in‚0$ memory_percent‚96.60453796386719$ proc.memory.availableMB‚556.3125! proc.memory.rssMB‚ 2814.421875* proc.memory.percent‚17.177867889404297 proc.cpu.threads‚17š|Ûú›Ò‚(: íÓêÔêáÌõ gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚126.02' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.203999999999997 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0È9(PWÓ‚(:Ì íÓêÔ‰µ‚ø$ disk.nvme0c0n1.out‚ 3482.01171875 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda.in‚0% disk./.usageGB‚1175.7823181152344" disk.nvme0n1.out‚ 3482.01171875 disk.dm-0.out‚ 3480.13671875 disk.sda.out‚ 5.5703125 disk.sda3.in‚0$ memory_percent‚96.60487174987793' proc.memory.availableMB‚ 556.2578125# proc.memory.rssMB‚ 2814.42578125 disk.drbd1260.out‚ 0.02734375 disk.dm-1.in‚0 disk.dm-1.out‚ 5.5703125 disk.drbd1260.in‚0 disk.dm-0.in‚ 471.640625* proc.memory.percent‚17.177891731262207 cpu‚25.545221946214735 proc.cpu.threads‚17 network.recv‚146910 disk.nvme0c0n1.in‚ 471.640625 disk.nvme0n1.in‚ 471.640625 disk.sda3.out‚ 5.5703125 network.sent‚182877' disk./.usagePercent‚78.907737915789 2¨0Ô+ _wandbruntime‚182  _runtime‚182ónÔ§ÂÕ¼ & lr-AdamW/pg1‚7.524336568891281e-05 & lr-AdamW/pg2‚7.524336568891281e-05 trainer/global_step‚99 % _timestamp‚1.7885209440746517e+09 _runtime‚ 182.294198884 _step‚4èŽ 7ÆÖÀ % _timestamp‚1.7885209440746517e+09 _runtime‚ 182.294198884 _step‚4 & lr-AdamW/pg1‚7.524336568891281e-05 & lr-AdamW/pg2‚7.524336568891281e-05 trainer/global_step‚99/œ0×+ _wandbruntime‚182  _runtime‚182Ðê-rñØ‘O JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚685 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 4141947826688 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 3946337505280 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1839991 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 199204256256 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytßÊ O¦#es.large_pool.freed‚ 3766624684240 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 285094912 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 300049920 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 4141696922112 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚2689298 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 4141947826688 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1839989 _runtime‚ 182.295541955 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 3946337505280 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 3766641723600 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚2664452 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 3946355508224 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 199489351168 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 _step‚5 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚2689298 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚699669 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 4185496465920 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚99 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1839991 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 3946355508224 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1283 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1788589 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 4384700722176 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 194805497804 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1088918 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 3961430182044 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 195359416832 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚683 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 4141696922112 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚2689298 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 4141947826688 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1839989 _runtime‚ 182.295541955 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 3946337505280 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 3766641723600 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚2664452 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 3946355508224 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 199489351168 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 _step‚5 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚2689298 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚699669 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 4185496465920 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚99 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1839991 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 3946355508224 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1283 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1788589 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 4384700722176 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 194805497804 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1088918 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 3961430182044 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 195359416832 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚683 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1839991 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 199204256256 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 3766624684240 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 285094912 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 300049920Ûè¿Y0Ú+ _wandbruntime‚183  _runtime‚183¤ Óv &ÛšM F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚4574282 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 4428685931520 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 4428376909824 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 4000919028454 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 3985623542272 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 4183197902848 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 3985623542272 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚705621 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚4549408 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 197556357632 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 201196473856 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 4182946984960 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 197323442688 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 196763986790 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚706309 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2691144 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚2716016 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 N6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2716016 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 197556357632 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1806031 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚690 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 3804155041664 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 3804172081024 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 196992960534 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 trainer/global_step‚99 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1858266 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1283 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1805341 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1099722 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 4183197902848 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1858266 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 _step‚6 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚4574282 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1858264 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 $ _timestamp‚1.788520945055035e+09 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚4549408 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1858264 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1099720 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚2691144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 300049920 _runtime‚ 183.275272076 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273“CI@Ì ÜžM SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1283 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1805341 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1099722 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 4183197902848 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1858266 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 _step‚6 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚4574282 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1858264 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 $ _timestamp‚1.788520945055035e+09 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚4549408 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1858264 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1099720 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚2691144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 300049920 _runtime‚ 183.275272076 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 F:DeviceStatsMoniÖ ÌâØtor.on_train_batch_end/active.all.allocated‚4574282 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 4428685931520 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 4428376909824 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 4000919028454 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 3985623542272 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 4183197902848 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 3985623542272 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚705621 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚4549408 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 197556357632 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 201196473856 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 4182946984960 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 197323442688 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 196763986790 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚706309 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2691144 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚2716016 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 N6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2716016 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 197556357632 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1806031 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚690 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 3804155041664 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 3804172081024 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 196992960534 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 trainer/global_step‚99 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1858266 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚4002404 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 5643142555440 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 292257792 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚6756143 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 293575123456 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 6204465382912 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 5910657357824 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 5910639354880 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 292972511848 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚9 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 6204214478336 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 5935886093752 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 5643159594800 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 292743538312 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 trainer/global_step‚149 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 6204214478336 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _runtime‚ 255.517287549 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚4002404 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 327892480 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 293808025088 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2992805222401×À–uï˜O ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 292972511848 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚9 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 6204214478336 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 5935886093752 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 5643159594800 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 292743538312 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 trainer/global_step‚149 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 6204214478336 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _runtime‚ 255.517287549 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚4002404 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 327892480 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 293808025088 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 SDDeviceñëbç)StatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 299280522240 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚4027250 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1048754 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 6568975490560 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚4002404 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 5643142555440 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 292257792 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚6756143 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 293575123456 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 6204465382912 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 5910657357824 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 5910639354880 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 þÆk0ð+ _wandbruntime‚256  _runtime‚256šzÊ¥&ñŸM PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚2772014 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1639822 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2695149 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 6311402878464 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 301275015168 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1639820 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 295539149312 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚10 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 5949925391872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 6612653953536 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 5975374940162 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 6245464541184 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 6612960878080 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚4029096 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚4053968 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1283 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 294702027298 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2695851 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 295772064256 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚6825984 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 6245715459072 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 295539149312 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306924544 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 NDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 5949943394816 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚6801110 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚4029096 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 301557999616 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 5680672912864 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚4053968 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚702 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚149 % _timestamp‚1.7885210182681994e+09 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2772014 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚6825984 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚700 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 327892480 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 5975620953266 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 294931001042 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚6801110 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 6311378938368 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1056029 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282984448 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2772016 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 5949943394816 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 5680689952224 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0üWÁ©&ò£M GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚2772014 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1639822 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2695149 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 6311402878464 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 301275015168 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1639820 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 295539149312 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚10 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 5949925391872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 6612653953536 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 5975374940162 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 6245464541184 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 6612960878080 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚4029096 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚4053968 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1283 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 294702027298 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2695851 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 295772064256 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚6825984 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 6245715459072 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 295539149312 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306924544 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 NDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 5949943394816 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚6801110 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚4029096 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 301557999616 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 5680672912864 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚4053968 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚702 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚149 % _timestamp‚1.7885210182681994e+09 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2772014 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚6825984 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚700 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 327892480 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 5975620953266 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 294931001042 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚6801110 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 6311378938368 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1056029 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282984448 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2772016 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 5949943394816 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 5680689952224 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 QÛà»0ó+ _wandbruntime‚256  _runtime‚256ΰÀLôÆ /loss/FRt2/assignment_loss‚7.116534172404896 1loss/SRqqt3/detection_loss‚3.6155837777516115 /loss/FRt4/detection_loss‚3.2487414152698313 epoch‚0 .loss/FRt2/detection_loss‚2.573751604015177 /loss/FRt4/assignment_loss‚7.498429399855594 1loss/SRqqt2/assignment_loss‚4.864813232421875 1loss/SRqqt3/assignment_loss‚5.313351071399191 1loss/SRqqt4/assignment_loss‚5.349620748073496 /loss/FRt3/assignment_loss‚7.321152329444885 .loss/FBt3/detection_loss‚3.861647753322469 0loss/FBt3/assignment_loss‚0.8488742172718048 0loss/SRqqt2/detection_loss‚2.308863132710879 trainer/global_step‚149 /loss/FRt1/assignment_loss‚7.197075952183116 1loss/FBt4/assignment_loss‚0.42498478552569513 /loss/FRt3/detection_loss‚3.9720563665032387 0loss/SRqqt1/detection_loss‚4.669751472984042 _runtime‚ 256.49250108 0loss/SRqqt1/assignment_loss‚5.19294696535383 0loss/FBt1/assignment_loss‚0.8670975863933563 -loss/FRt1/detection_loss‚6.19822885908867 _step‚11 0loss/FBt2/assignment_loss‚0.4845311729156453 1loss/SRqqt4/detection_loss‚2.7915912537589085 .loss/FBt2/detection_loss‚3.479466962425605 &loss/total_loss‚110.44051303318572 /loss/FBt1/detection_loss‚13.624921801189581 .loss/FBt4/detection_loss‚7.616497000922328 % _timestamp‚1.7885210182727711e+09¤ks´PõÊ 0loss/SRqqt1/assignment_loss‚5.19294696535383 0loss/FBt1/assignment_loss‚0.8670975863933563 -loss/FRt1/detection_loss‚6.19822885908867 _step‚11 0loss/FBt2/assignment_loss‚0.4845311729156453 1loss/SRqqt4/detection_loss‚2.7915912537589085 .loss/FBt2/detection_loss‚3.479466962425605 &loss/total_loss‚110.44051303318572 /loss/FBt1/detection_loss‚13.624921801189581 .loss/FBt4/detection_loss‚7.616497000922328 % _timestamp‚1.7885210182727711e+09 /loss/FRt2/assignment_loss‚7.116534172404896 1loss/SRqqt3/detection_loss‚3.6155837777516115 /loss/FRt4/detection_loss‚3.2487414152698313 epoch‚0 .loss/FRt2/detection_loss‚2.573751604015177 /loss/FRt4/assignment_loss‚7.498429399855594 1loss/SRqqt2/assignment_loss‚4.864813232421875 1loss/SRqqt3/assignment_loss‚5.313351071399191 1loss/SRqqt4/assignment_loss‚5.349620748073496 /loss/FRt3/assignment_loss‚7.321152329444885 .loss/FBt3/detection_loss‚3.861647753322469 0loss/FBt3/assignment_loss‚0.8488742172718048 0loss/SRqqt2/detection_loss‚2.308863132710879 trainer/global_step‚149 /loss/FRt1/assignment_loss‚7.197075952183116 1loss/FBt4/assignment_loss‚0.42498478552569513 /loss/FRt3/detection_loss‚3.9720563665032387 0loss/SRqqt1/detection_loss‚4.669751472984042 _runtime‚ 256.49250108 y·‘ö‚(:† ÇÔêÔ×ɲø gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚109.102' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.8204 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÃOõèZ÷‚(:Ï ÇÔêÔÁïáø network.recv‚176515 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda1.in‚0 disk.drbd1260.in‚0! proc.memory.rssMB‚ 2814.453125* proc.memory.percent‚17.178058624267578 network.sent‚262768 disk.sda.out‚ 9.23046875 disk.sda3.out‚ 9.23046875 disk.nvme0n1.in‚ 708.40625 disk.dm-1.in‚0 disk.dm-1.out‚ 9.23046875 disk.drbd1260.out‚ 0.0390625 proc.cpu.threads‚17 disk.nvme0c0n1.in‚ 708.40625% disk.nvme0c0n1.out‚5404.560546875) disk./.usagePercent‚78.79861368028894% disk./.usageGB‚1174.1562881469727 disk.dm-0.out‚5404.560546875 disk.sda.in‚0 disk.sda1.out‚0$ memory_percent‚96.58515453338623 disk.dm-0.in‚ 708.40625# disk.nvme0n1.out‚5404.560546875( proc.memory.availableMB‚ 559.48828125 cpu‚25.421895755238932]Ð]^ø‚(:Ó ÖÔêÔ÷ò¾÷( disk./.usagePercent‚78.8039017719162% disk./.usageGB‚1174.2350845336914 disk.sda3.in‚0 disk.sda3.out‚9.609375! disk.nvme0c0n1.in‚ 718.1796875 disk.nvme0n1.in‚ 718.1796875 disk.sda1.out‚0( proc.memory.availableMB‚ 559.42578125 network.sent‚311948 disk.sda2.in‚0 disk.sda2.out‚0" disk.dm-0.out‚5485.82470703125 disk.sda.in‚0 disk.sda1.in‚0# memory_percent‚96.5855360031128! proc.memory.rssMB‚ 2814.453125' disk.nvme0c0n1.out‚5485.82470703125 disk.dm-1.out‚9.609375* proc.memory.percent‚17.178058624267578 proc.cpu.threads‚17 network.recv‚181599 disk.dm-0.in‚ 718.1796875 disk.sda.out‚9.609375% disk.nvme0n1.out‚5485.82470703125 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.04296875 cpu‚25.40847535832042g>‘ù‚(:… ÖÔêÔŠç£ù gpu.0.gpu‚8.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚105.117' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.0234 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Çص0‘ú‚(:† åÔêÔ›¦¸ø gpu.0.gpu‚26.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚122.648' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.5296 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0*—£Åfû‚(:Û åÔêÔ„‘íø disk.dm-0.in‚ 718.1796875" disk.dm-0.out‚5744.98486328125 network.sent‚318054! disk.nvme0c0n1.in‚ 718.1796875 disk.dm-1.in‚0* proc.memory.percent‚17.178058624267578 cpu‚25.401544215302764 proc.cpu.threads‚17 network.recv‚186005% disk./.usageGB‚1174.2350273132324 disk.sda3.out‚ 10.0390625% disk.nvme0n1.out‚5744.98486328125 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda.out‚ 10.0390625 disk.sda3.in‚0' disk.nvme0c0n1.out‚5744.98486328125 disk.sda1.in‚0$ memory_percent‚96.58410549163818( proc.memory.availableMB‚ 559.66015625! proc.memory.rssMB‚ 2814.453125( disk./.usagePercent‚78.8038979318032 disk.nvme0n1.in‚ 718.1796875 disk.dm-1.out‚ 10.0390625 disk.drbd1260.out‚ 0.04296875 disk.sda2.in‚0 disk.sda2.out‚0¶TH9`ü‚(:Õ ôÔêÔ¹ýàø) disk./.usagePercent‚78.80453513455592 disk.sda2.in‚0 disk.dm-0.in‚ 718.1796875 disk.sda.out‚ 10.640625 disk.sda3.in‚0 disk.sda1.in‚0" disk.dm-0.out‚5947.96044921875! disk.nvme0c0n1.in‚ 718.1796875% disk.nvme0n1.out‚5947.96044921875 disk.dm-1.out‚ 10.640625 disk.drbd1260.out‚0.046875 proc.cpu.threads‚17 network.sent‚326597 disk.sda.in‚0 disk.nvme0n1.in‚ 718.1796875$ memory_percent‚96.58358097076416( proc.memory.availableMB‚ 559.74609375* proc.memory.percent‚17.178058624267578% disk./.usageGB‚1174.2445220947266 disk.sda2.out‚0' disk.nvme0c0n1.out‚5947.96044921875 disk.sda1.out‚0 disk.dm-1.in‚0 disk.drbd1260.in‚0! proc.memory.rssMB‚ 2814.453125 cpu‚25.3932602335509 disk.sda3.out‚ 10.640625 network.recv‚1915536D¼¢›ý‚(: ôÔêÔ¦îÅù gpu.0.gpu‚13.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚99.556' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚19.911199999999997 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÏŸ"Ÿþ‚(:… ƒÕêÔ®ú÷ö gpu.0.gpu‚29.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚122.935' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.587 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÒòÅ›`ÿ‚(:Õ ƒÕêÔïøñ÷% disk.nvme0n1.out‚6231.08154296875 disk.drbd1260.in‚0% proc.memory.availableMB‚ 559.46875 network.sent‚330256$ disk./.usageGB‚1174.244541168213 disk.dm-0.in‚ 718.1796875' disk.nvme0c0n1.out‚6231.08154296875 disk.nvme0n1.in‚ 718.1796875 disk.sda2.in‚0 disk.sda2.out‚0! proc.memory.rssMB‚ 2814.453125 disk.sda.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 10.9921875 disk.sda1.out‚0 disk.drbd1260.out‚0.046875* proc.memory.percent‚17.178058624267578 cpu‚25.388396914827446 proc.cpu.threads‚17 network.recv‚195607" disk.dm-0.out‚6231.08154296875! disk.nvme0c0n1.in‚ 718.1796875 disk.sda1.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 10.9921875$ memory_percent‚96.58527374267578( disk./.usagePercent‚78.8045364145936 disk.sda.out‚ 10.9921875ɨۚ0€+ _wandbruntime‚333  _runtime‚333‰Ù®À ' lr-AdamW/pg1‚0.009j¼©015124676537468334 ' lr-AdamW/pg2‚0.00015124676537468334 trainer/global_step‚199 % _timestamp‚1.7885210948135617e+09 _runtime‚ 333.033338929 _step‚12DÈ‘Ê‚Ä ' lr-AdamW/pg2‚0.00015124676537468334 trainer/global_step‚199 % _timestamp‚1.7885210948135617e+09 _runtime‚ 333.033338929 _step‚12 ' lr-AdamW/pg1‚0.00015124676537468334 fµP!0ƒ+ _wandbruntime‚333  _runtime‚333#¶÷™'„“O N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3569715 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 390681578820 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 # _timestamp‚1.78852109481501e+09 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚9007845 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 7910588018356 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1399926 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 _step‚13 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 392023731712 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 304840704 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 300049920 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 392023731712 trainer/global_step‚199 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 391790830080 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3667489 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 8353893470720 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 399754376192 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 7874959207424 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 8753366946304 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 390910552356 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚3667489 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 7874959207424 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3667491 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 7519660426640 _runtime‚ 333.034577573 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 399473475584 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 8266732034560 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 391790830080 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 7874941204480 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 8753671787008 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 7910342005460 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚5340356 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1283 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚671 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2169118 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚9032693 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 327892480 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 8353917410816 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚3667491 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 8266982939136 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚669 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 8266982939136 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 8266732034560 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1302 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2169120 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1400595 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚5365202 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 280900608 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 7874941204480 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚5340356 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3569044 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0û[¾}'…—O LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3569715 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 390681578820 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 # _timestamp‚1.78852109481501e+09 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚9007845 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 7910588018356 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1399926 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 _step‚13 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 392023731712 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 304840704 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 300049920 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 392023731712 trainer/global_step‚199 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 391790830080 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3667489 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 8353893470720 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 399754376192 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 7874959207424 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 8753366946304 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 390910552356 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚3667489 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 7874959207424 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3667491 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 7519660426640 _runtime‚ 333.034577573 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 399473475584 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 8266732034560 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 391790830080 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 7874941204480 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 8753671787008 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 7910342005460 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚5340356 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1283 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚671 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2169118 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚9032693 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 327892480 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 8353917410816 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚3667491 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 8266982939136 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚669 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 8266982939136 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 8266732034560 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1302 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2169120 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1400595 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚5365202 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 280900608 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 7874941204480 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚5340356 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3569044 >‹æâ0†+ _wandbruntime‚334  _runtime‚334ôÖ­¤&‡žM QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2179920 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 _step‚14 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3685764 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 8307982097408 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1302 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 % _timestamp‚1.7885210958077185e+09 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚9077686 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1283 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3586929 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 285081600 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 7949830851870 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 393754855936 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 7914227241472 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 7914245244416 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 309021696 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 8308233015296 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3685766 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 8797046302208 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 trainer/global_step‚199 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3685764 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3587652 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 392640067806 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3685766 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 7557207823424 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 7914227241472 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 8395577440768 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚5367048 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 393987770880 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 393754855936 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 392869041550 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚5367048 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚9052812 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚723 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1407730 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0käIˆ¢M B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚9052812 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚723 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1407730 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2179920 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 _step‚14 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3685764 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 8307982097408 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1302 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 % _time 7Ñ…stamp‚1.7885210958077185e+09 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚9077686 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1283 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3586929 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 285081600 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 7949830851870 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 393754855936 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 7914227241472 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 7914245244416 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 309021696 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 8308233015296 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3685766 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 8797046302208 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 trainer/global_step‚199 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3685764 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3587652 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 392640067806 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3685766 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 7557207823424 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 7914227241472 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 8395577440768 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚5367048 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 393987770880 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 393754855936 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 392869041550 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚5367048 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309îêÒÚ0‰+ _wandbruntime‚334  _runtime‚334ñ&(ôRŠÌ 2loss/SRqqt2/assignment_loss‚3.4484734953495493 2loss/SRqqt3/assignment_loss‚3.5801808086793816 0loss/FBt3/assignment_loss‚0.6456763554036755 /loss/FRt2/detection_loss‚2.8097708871201523 trainer/global_step‚199 /loss/FRt4/assignment_loss‚7.445840719269543 1loss/SRqqt1/assignment_loss‚3.476276091166905 1loss/FBt2/assignment_loss‚0.09630324549895401 .loss/FRt1/detection_loss‚3.945690851520912 /loss/FRt1/assignment_loss‚7.179186644377531 2loss/SRqqt4/assignment_loss‚3.8369615392864875 .loss/FRt3/detection_loss‚2.936387154853569 1loss/SRqqt3/detection_loss‚2.2528845625582026 0loss/SRqqt1/detection_loss‚4.782976912600653 1loss/SRqqt4/detection_loss‚1.9019934077507472 .loss/FBt2/detection_loss‚7.851326819509268 /loss/FBt3/detection_loss‚1.1634151978955574 _runtime‚ 334.030735744 /loss/FRt3/assignment_loss‚7.273805114458192 /loss/FBt1/assignment_loss‚0.491107573111852 .loss/FRt4/detection_loss‚4.036745131742664 % _timestamp‚1.7885210958112922e+09 _step‚15 /loss/FRt2/assignment_loss‚7.232282458611254 .loss/FBt1/detection_loss‚26.87028117974599 1loss/FBt4/assignment_loss‚0.43344490915875544 0loss/SRqqt2/detection_loss‚2.472993553915059 /loss/FBt4/detection_loss‚15.989445835351944 &loss/total_loss‚122.15345044893681 epoch‚0ŸÿÕV‹Ð _step‚15 /loss/FRt2/assignment_loss‚7.232282458611254 .loss/FBt1/detection_loss‚26.87028117974599 1loss/FBt4/assignment_loss‚0.43344490915875544 0loss/SRqqt2/detection_loss‚2.472993553915059 /loss/FBt4/detection_loss‚15.989445835351944 &loss/total_loss‚122.15345044893681 epoch‚0 2loss/SRqqt2/assignment_loss‚3.4484734953495493 2loss/SRqqt3/assignment_loss‚3.5801808086793816 0loss/FBt3/assignment_loss‚0.6456763554036755 /loss/FRt2/detection_loss‚2.8097708871201523 trainer/global_step‚199 /loss/FRt4/assignment_loss‚7.445840719269543 1loss/SRqqt1/assignment_loss‚3.476276091166905 1loss/FBt2/assignment_loss‚0.09630324549895401 .loss/FRt1/detection_loss‚3.945690851520912 /loss/FRt1/assignment_loss‚7.179186644377531 2loss/SRqqt4/assignment_loss‚3.8369615392864875 .loss/FRt3/detection_loss‚2.936387154853569 1loss/SRqqt3/detection_loss‚2.2528845625582026 0loss/SRqqt1/detection_loss‚4.782976912600653 1loss/SRqqt4/detection_loss‚1.9019934077507472 .loss/FBt2/detection_loss‚7.851326819509268 /loss/FBt3/detection_loss‚1.1634151978955574 _runtime‚ 334.030735744 /loss/FRt3/assignment_loss‚7.273805114458192 /loss/FBt1/assignment_loss‚0.491107573111852 .loss/FRt4/detection_loss‚4.036745131742664 % _timestamp‚1.7885210958112922e+09íãþ¹Œ‚(:„ ’ÕêÔ™ðþö gpu.0.gpu‚12.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚98.675' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.735 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0˜vä"Z‚(:Ï ’ÕêÔËûÐø disk.sda2.out‚0 disk.dm-0.in‚ 861.359375 disk.nvme0n1.in‚ 861.359375 disk.sda1.in‚0 cpu‚25.38179661955692 proc.cpu.threads‚17) disk./.usagePercent‚78.80453513455592% disk./.usageGB‚1174.2445220947266! disk.dm-0.out‚6424.0732421875 disk.sda3.in‚0& disk.nvme0c0n1.out‚6424.0732421875 disk.dm-1.in‚0 disk.dm-1.out‚11.53125( proc.memory.availableMB‚ 559.62890625 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.05078125 disk.sda.out‚11.53125 disk.sda3.out‚11.53125 disk.nvme0c0n1.in‚ 861.359375$ memory_percent‚96.58429622650146* proc.memory.percent‚17.178058624267578 network.recv‚199229 disk.sda2.in‚0 disk.sda.in‚0$ disk.nvme0n1.out‚6424.0732421875 disk.sda1.out‚0! proc.memory.rssMB‚ 2814.453125 network.sent‚335730ý k¿Ž‚(:„ ¡ÕêÔøà ÷ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚92.922' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.5844 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ðå·d‚(:Ù ¡ÕêÔþŽø cpu‚25.372192116440438 proc.cpu.threads‚17) disk./.usagePercent‚78.80872546587636 disk.nvme0n1.in‚ 861.359375 disk.dm-1.in‚0 disk.dm-1.out‚ 11.99609375 disk.drbd1260.in‚0 disk.dm-0.in‚ 861.359375 disk.sda.in‚0 disk.sda3.out‚ 11.99609375 network.sent‚385626 disk.sda1.in‚0 disk.sda.out‚ 11.99609375 disk.nvme0c0n1.in‚ 861.359375$ memory_percent‚96.58305644989014( proc.memory.availableMB‚ 559.83203125! proc.memory.rssMB‚ 2814.453125* proc.memory.percent‚17.178058624267578$ disk.nvme0n1.out‚6572.8466796875 disk.sda1.out‚0 disk.sda3.in‚0& disk.nvme0c0n1.out‚6572.8466796875 network.recv‚205378% disk./.usageGB‚1174.3069610595703 disk.drbd1260.out‚ 0.05078125 disk.sda2.in‚0 disk.sda2.out‚0! disk.dm-0.out‚6572.84667968753–¿e‘‚(:† °ÕêÔáäˆú gpu.0.gpu‚22.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚102.956' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.5912 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0{Uéºd‘‚(:Ù °ÕêÔƒÉÿ network.recv‚210480 disk.nvme0c0n1.in‚ 861.359375' disk.nvme0c0n1.out‚6808.64306640625 disk.nvme0n1.in‚ 861.359375 disk.sda1.out‚0 disk.dm-1.in‚0! proc.memory.rssMB‚ 2814.453125 disk.dm-1.out‚ 12.46484375 disk.drbd1260.in‚0 disk.dm-0.in‚ 861.359375 disk.sda.in‚0 disk.sda.out‚ 12.46484375 disk.sda3.in‚0 disk.drbd1260.out‚ 0.0546875$ disk./.usageGB‚1174.369728088379 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0$ memory_percent‚96.58169746398926' proc.memory.availableMB‚ 560.0546875 cpu‚25.363511023517894 network.sent‚392178) disk./.usagePercent‚78.81293781384475% disk.nvme0n1.out‚6808.64306640625" disk.dm-0.out‚6808.64306640625 disk.sda3.out‚ 12.46484375* proc.memory.percent‚17.178058624267578 proc.cpu.threads‚17ÃE¶’’‚(:… ¿ÕêÔæÌäõ gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚110.595' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.119 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µ¸?e“‚(:Ú ¿ÕêÔÿÉØø disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.0546875 disk.sda2.in‚0! proc.memory.rssMB‚ 2814.453125 network.recv‚214211 disk.sda.in‚0 disk.sda1.in‚0' proc.memory.availableMB‚ 559.8046875) disk./.usagePercent‚78.81292117335504 disk.sda.out‚ 12.88671875 disk.nvme0c0n1.in‚ 861.359375' disk.nvme0c0n1.out‚7011.00244140625% disk.nvme0n1.out‚7011.00244140625 disk.dm-1.out‚ 12.88671875 disk.sda2.out‚0* proc.memory.percent‚17.178058624267578 disk.dm-0.in‚ 861.359375" disk.dm-0.out‚7011.00244140625 disk.sda3.out‚ 12.88671875 disk.sda1.out‚0 disk.dm-1.in‚0$ memory_percent‚96.58322334289551 cpu‚25.358147198732407 proc.cpu.threads‚17 network.sent‚397580% disk./.usageGB‚1174.3694801330566 disk.sda3.in‚0 disk.nvme0n1.in‚ 861.359375G(ó8”‚(:„ ÎÕêÔ“ù‰÷ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.913' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.7826 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0@ÿøe•‚(:Ú ÎÕêÔ×̵ø disk.dm-1.in‚0 disk.dm-1.out‚ 13.2265625 disk.sda.out‚ 13.2265625$ memory_percent‚96.58181667327881) disk./.usagePercent‚78.82131412435757' disk.nvme0c0n1.out‚7261.80810546875 disk.sda1.in‚0 disk.drbd1260.in‚0 disk.dm-0.in‚ 992.3828125( proc.memory.availableMB‚ 560.03515625 cpu‚25.350236920621178 proc.cpu.threads‚17$ disk./.usageGB‚1174.494541168213! disk.nvme0c0n1.in‚ 992.3828125% disk.nvme0n1.out‚7261.80810546875 disk.sda2.in‚0 disk.sda.in‚0" proc.memory.rssMB‚ 2814.4609375( proc.memory.percent‚17.1781063079834 disk.sda3.in‚0 disk.sda3.out‚ 13.2265625 disk.nvme0n1.in‚ 992.3828125 disk.sda1.out‚0 disk.drbd1260.out‚ 0.05859375 disk.sda2.out‚0" disk.dm-0.out‚7261.80810546875 network.sent‚402133 network.recv‚217425 Dú–‚(:„ ÝÕêÔÊЊø gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚96.211' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.2422 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0QÅb—‚(:× ÝÕêÔÍÐçù( proc.memory.percent‚17.1781063079834 proc.cpu.threads‚17) disk./.usagePercent‚78.82130951622194 disk.sda3.in‚0 disk.sda3.out‚ 13.56640625 disk.sda1.in‚0 disk.sda1.out‚0 cpu‚25.342644691656062 disk.dm-0.in‚ 992.3828125" disk.dm-0.out‚7602.90966796875 disk.sda.in‚0 disk.sda.out‚ 13.56640625! disk.nvme0c0n1.in‚ 992.3828125' disk.nvme0c0n1.out‚7602.90966796875 disk.nvme0n1.in‚ 992.3828125 disk.drbd1260.in‚0 network.sent‚408436 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 13.56640625 disk.drbd1260.out‚ 0.05859375$ memory_percent‚96.58355712890625" proc.memory.availableMB‚559.75" proc.memory.rssMB‚ 2814.4609375 network.recv‚222650$ disk./.usageGB‚1174.494472503662% disk.nvme0n1.out‚7602.90966796875 disk.dm-1.in‚0¸#0˜+ _wandbruntime‚430  _runtime‚430 ïÄ™¾ & lr-AdamW/pg1‚0.0001892484652175686 & lr-AdamW/pg2‚0.0001892484652175686 trainer/global_step‚249 % _timestamp‚1.7885211924861915e+09 _runtime‚ 430.706573759 _step‚16íÍ]Ț & lr-AdamW/pg1‚0.0001892484652175686 & lr-AdamW/pg2‚0.0001892484652175686 trainer/global_step‚249 % _timestamp‚1.7885211924861915e+09 _runtime‚ 430.706573759 _step‚16 ¨«0›+ _wandbruntime‚430  _runtime‚430­æ¶¦'œ O LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 $ _timestamp‚1.788521192488164e+09 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2709220 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 9884797917168 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 490006536704 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚4581239 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 _step‚17 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1285 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 490239438336 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 490006536704 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚10329249590784 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1749213 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 488619619328 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚661 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4458431 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 _runtime‚ 430.708175105 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2709218 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚6703154 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 9839261057024 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 300049920 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 9396195337200 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 trainer/global_step‚249 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1749872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚6678308 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚10329500495360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4459092 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚10438091973120 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚11284395 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚659 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚10329249590784 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0^%ݖĤO $ _timestamp‚1.788521192488164e+09 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2709220 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 9884797917168 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 490006536704 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚4581239 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 _step‚17 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1285 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 490239438336 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 490006536704 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚10329249590784 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1749213 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 488619619328 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚661 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4458431 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 _runtime‚ 430.708175105 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2709218 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚6703154 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 9839261057024 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 300049920 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 9396195337200 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 trainer/global_step‚249 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1749872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚6678308 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚10329500495360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4459092 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚10438091973120 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚11284395 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚659 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚10329249590784 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309zKþ&0ž+ _wandbruntime‚432  _runtime‚432ZÃQ±&Ÿ«M ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚10479799883264 _step‚18 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2720020 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚11304514 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 9878547094016 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚4599516 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 490807082058 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚ 432.097244723 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4599514 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚11304514 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1285 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 9433725694624 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6705000 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 9924532776682 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 491970562560 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4476166 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 294341632 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 9924286763578 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 trainer/global_step‚249 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4599516 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 9878529091072 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1304 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚10479775943168 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚10370750571520 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1756782 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚6729872 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚6729872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7885211938721106e+09 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11329388 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚636 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚10981328357888 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 9878529091072 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1304 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚10479775943168 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚10370750571520 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1756782 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚6729872 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚6729872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7885211938721106e+09 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11329388 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚636 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚10981328357888 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚ 432.097244723 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4599514 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚11304514 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1285 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 9433725694624 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6705000 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 9924532776682 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 491970562560 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4476166 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 294341632 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 9924286763578 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 trainer/global_step‚249 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104•Ñ¤}0¡+ _wandbruntime‚432  _runtime‚432Ç£T¢Î .loss/FBt2/detection_loss‚2.297992385863372 2loss/SRqqt4/assignment_loss‚3.9559645739468663 .loss/FRt4/detection_loss‚3.952139010032018 1loss/SRqqt3/detection_loss‚3.0813177502155304 2loss/SRqqt3/assignment_loss‚3.2473990631103518 0loss/FBt1/assignment_loss‚0.5660037708214738 $ _timestamp‚1.788521193880756e+09 epoch‚0 .loss/FRt1/detection_loss‚4.642643623731353 1loss/SRqqt2/assignment_loss‚3.313423679155462 1loss/FBt4/assignment_loss‚0.48405349692038724 _runtime‚ 432.101410036 /loss/FRt2/assignment_loss‚7.214505680820398 /loss/FRt3/assignment_loss‚7.267057323455811 /loss/FRt2/detection_loss‚2.5818089253285477 1loss/SRqqt2/detection_loss‚1.7607908216797274 /loss/FBt1/detection_loss‚14.004374566403301 trainer/global_step‚249 /loss/FRt4/assignment_loss‚7.360107558114188 /loss/FBt3/assignment_loss‚0.787522364230383 1loss/SRqqt1/detection_loss‚5.9713880684640674 1loss/SRqqt4/detection_loss‚2.4007139094664707 &loss/total_loss‚100.83104352910063 _step‚19 /loss/FBt3/detection_loss‚3.4835124227798695 .loss/FBt4/detection_loss‚8.257280772924423 0loss/FRt1/assignment_loss‚7.2476067326285625 2loss/SRqqt1/assignment_loss‚3.3417701897797762 1loss/FBt2/assignment_loss‚0.39963251005232736 .loss/FRt3/detection_loss‚3.2120343291759496ŸÏcX£Ò 2loss/SRqqt3/assignment_loss‚3.2473990631103518 0loss/FBt1/assignment_loss‚0.5660037708214738 $ _timestamp‚1.788521193880756e+09 epoch‚0 .loss/FRt1/detection_loss‚4.642643623731353 1loss/SRqqt2/assignment_loss‚3.313423679155462 1loss/FBt4/assignment_loss‚0.48405349692038724 _runtime‚ 432.101410036 /loss/FRt2/assignment_loss‚7.214505680820398 /loss/FRt3/assignment_loss‚7.267057323455811 /loss/FRt2/detection_loss‚2.5818089253285477 1loss/SRqqt2/detection_loss‚1.7607908216797274 /loss/FBt1/detection_loss‚14.004374566403301 trainer/global_step‚249 /loss/FRt4/assignment_loss‚7.360107558114188 /loss/FBt3/assignment_loss‚0.787522364230383 1loss/SRqqt1/detection_loss‚5.9713880684640674 1loss/SRqqt4/detection_loss‚2.4007139094664707 &loss/total_loss‚100.83104352910063 _step‚19 /loss/FBt3/detection_loss‚3.4835124227798695 .loss/FBt4/detection_loss‚8.257280772924423 0loss/FRt1/assignment_loss‚7.2476067326285625 2loss/SRqqt1/assignment_loss‚3.3417701897797762 1loss/FBt2/assignment_loss‚0.39963251005232736 .loss/FRt3/detection_loss‚3.212034329175949 .loss/FBt2/detection_loss‚2.297992385863372 2loss/SRqqt4/assignment_loss‚3.9559645739468663 .loss/FRt4/detection_loss‚3.952139010032018 1loss/SRqqt3/detection_loss‚3.0813177502155304Z _}^¤‚(:Ó ìÕêÔõÓåù$ memory_percent‚96.58262729644775( proc.memory.percent‚17.1781063079834! disk.nvme0c0n1.in‚ 992.3828125 disk.nvme0n1.in‚ 992.3828125 disk.sda2.out‚0 disk.dm-0.out‚7886.255859375 disk.sda3.out‚ 14.06640625 proc.cpu.threads‚17 network.sent‚415037# disk.nvme0n1.out‚7886.255859375 disk.sda1.out‚0 disk.dm-1.out‚ 14.06640625 disk.sda.in‚0 disk.sda.out‚ 14.06640625 disk.sda3.in‚0( proc.memory.availableMB‚ 559.90234375) disk./.usagePercent‚78.83816300421606 disk.dm-1.in‚0 disk.drbd1260.out‚0.0625 disk.sda2.in‚0" proc.memory.rssMB‚ 2814.4609375 cpu‚25.33516396214375 disk.drbd1260.in‚0 network.recv‚227821% disk./.usageGB‚1174.7456016540527% disk.nvme0c0n1.out‚7886.255859375 disk.sda1.in‚0 disk.dm-0.in‚ 992.3828125¸(,•¥‚(:„ ìÕêÔž­…ú gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚92.922' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.5844 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¶®Jn‘¦‚(:† ûÕêÔÜŒÄö gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚111.396' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.2792 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0§S.c§‚(:Ø ûÕêÔô—š÷" disk.dm-0.out‚7944.69287109375 disk.sda3.out‚ 14.44140625 disk.nvme0n1.in‚ 992.3828125 disk.drbd1260.out‚0.0625$ memory_percent‚96.58288955688477& proc.memory.availableMB‚ 559.859375 network.sent‚463850 disk.sda.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 14.44140625 disk.sda2.out‚0( proc.memory.percent‚17.1781063079834 cpu‚25.329392716346657 network.recv‚232669) disk./.usagePercent‚78.83796587841469 disk.sda.out‚ 14.44140625' disk.nvme0c0n1.out‚7944.69287109375 disk.dm-1.in‚0 proc.cpu.threads‚17 disk.sda3.in‚0! disk.nvme0c0n1.in‚ 992.3828125% disk.nvme0n1.out‚7944.69287109375 disk.drbd1260.in‚0 disk.sda2.in‚0" proc.memory.rssMB‚ 2814.4609375% disk./.usageGB‚1174.7426643371582 disk.dm-0.in‚ 992.3828125£Wßœ¨‚(:‘ ŠÖêÔôþóõ gpu.0.gpu‚30.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚111.362' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.272399999999998 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0z $­Ð©‚(:É ŠÖêÔ¿ŒÇø disk.sda3.out‚ 14.83203125 disk.drbd1260.out‚ 0.06640625 cpu‚25.32404200786371 proc.cpu.threads‚17 disk.sda2.in‚0 disk.dm-0.in‚1142.875 disk.sda.out‚ 14.83203125$ disk.nvme0c0n1.out‚ 8166.44140625 disk.sda1.out‚0 network.recv‚236966 disk.sda2.out‚0 disk.sda.in‚0 disk.sda3.in‚0 disk.sda1.in‚0 disk.dm-1.in‚0( proc.memory.availableMB‚ 559.84765625% disk./.usageGB‚1174.7426147460938 disk.nvme0n1.in‚1142.875" disk.nvme0n1.out‚ 8166.44140625 disk.dm-1.out‚ 14.83203125 disk.drbd1260.in‚0# memory_percent‚96.5829610824585 proc.memory.rssMB‚ 2814.46875) proc.memory.percent‚17.17815399169922`û*2„ disk.nvme0c0n1.in‚1142.875 network.sent‚469517) disk./.usagePercent‚78.83796255031675 disk.dm-0.out‚ 8166.44140625é(xª‚(:„ ™ÖêÔºžÿö gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚29.0 gpu.0.powerWatts‚92.744' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.5488 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0# cÓQ«‚(:Æ ™ÖêÔ³Íâø disk.sda2.out‚0 disk.sda.in‚0 disk.nvme0n1.in‚1142.875$ memory_percent‚96.57297134399414 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.nvme0c0n1.in‚1142.875" disk.nvme0n1.out‚ 8477.03515625 disk.dm-1.out‚ 15.1484375& proc.memory.availableMB‚ 561.484375 proc.memory.rssMB‚ 2814.46875) proc.memory.percent‚17.17815399169922 disk.sda3.in‚0 network.sent‚475780 disk.dm-0.in‚1142.875 disk.dm-0.out‚ 8477.03515625$ disk.nvme0c0n1.out‚ 8477.03515625 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0% disk./.usageGB‚1174.7210731506348 disk.sda.out‚ 15.1484375 cpu‚25.318164389638255 proc.cpu.threads‚17 disk.sda3.out‚ 15.1484375 network.recv‚241514) disk./.usagePercent‚78.83651687577088 disk.drbd1260.out‚ 0.06640625 è*Ü‘¬‚(:† ¨ÖêÔÅ‚¬÷ gpu.0.gpu‚18.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚102.422' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.4844 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0½6(Y­‚(:Î ¨ÖêÔä¿Æú disk.sda3.out‚ 15.70703125) proc.memory.percent‚17.17815399169922 proc.cpu.threads‚17 network.sent‚482332 network.recv‚246558$ disk./.usageGB‚1174.744514465332 disk.drbd1260.out‚ 0.0703125 disk.sda2.in‚0 disk.sda.in‚0 disk.sda.out‚ 15.70703125 disk.sda3.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.in‚0" disk.dm-0.out‚8806.46240234375$ memory_percent‚96.57459259033203% proc.memory.availableMB‚ 561.21875 proc.memory.rssMB‚ 2814.46875( disk./.usagePercent‚78.8380900420688 disk.nvme0c0n1.in‚1142.875' disk.nvme0c0n1.out‚8806.46240234375 disk.nvme0n1.in‚1142.875% disk.nvme0n1.out‚8806.46240234375 disk.dm-1.in‚0 cpu‚25.312917907086295 disk.dm-1.out‚ 15.70703125 disk.sda2.out‚0 disk.dm-0.in‚1142.875tí´k®‚(:… ·ÖêÔ€øÛõ gpu.0.gpu‚27.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚134.515' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.903 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µ­—ƒZ¯‚(:Ï ·ÖêÔÝÛÎ÷" disk.dm-0.out‚9022.95068359375( proc.memory.availableMB‚ 561.45703125 cpu‚25.30794540189854 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.out‚ 0.0703125 disk.sda.in‚0 disk.sda3.out‚ 16.0546875 disk.nvme0c0n1.in‚1142.875% disk.nvme0n1.out‚9022.95068359375 proc.memory.rssMB‚ 2814.46875% disk./.usageGB‚1174.7445335388184 disk.dm-1.in‚0 disk.sda2.out‚0 disk.nvme0n1.in‚1142.875) proc.memory.percent‚17.17815399169922 proc.cpu.threads‚17) disk./.usagePercent‚78.83809132210646 disk.dm-1.out‚ 16.0546875 disk.drbd1260.in‚0 disk.dm-0.in‚1142.875 disk.sda.out‚ 16.0546875 disk.sda3.in‚0' disk.nvme0c0n1.out‚9022.95068359375$ memory_percent‚96.57313823699951 network.sent‚487691 network.recv‚250598 disk.sda2.in‚0"Â1Bœ°‚(:‘ ÆÖêÔ÷Áƒö gpu.0.gpu‚26.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚105.356' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.071199999999997 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ü­àÛe±‚(:Ú ÆÖêԾŧø network.sent‚493311 disk.dm-1.out‚ 16.41015625 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.07421875 disk.sda2.in‚0 disk.dm-0.in‚ 1285.92578125 disk.sda.out‚ 16.41015625 disk.sda3.out‚ 16.41015625 disk.dm-0.out‚ 9198.73828125 disk.sda3.in‚0! disk.nvme0n1.in‚ 1285.92578125 disk.sda1.in‚0 proc.memory.rssMB‚ 2814.46875 cpu‚25.302902561219923 proc.cpu.threads‚17 network.recv‚254946% disk./.usageGB‚1174.9893341064453 disk.dm-1.in‚0 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 1285.92578125$ disk.nvme0c0n1.out‚ 9198.73828125 disk.sda1.out‚0) disk./.usagePercent‚78.85452009360127 disk.sda.in‚0" disk.nvme0n1.out‚ 9198.73828125$ memory_percent‚96.57471179962158( proc.memory.availableMB‚ 561.19921875) proc.memory.percent‚17.17815399169922%ÄK80²+ _wandbruntime‚529  _runtime‚529CdÕƳÀ _step‚20 ' lr-AdamW/pg1‚0.00022725016506045388 ' lr-AdamW/pg2‚0.00022725016506045388 trainer/global_step‚299 % _timestamp‚1.7885212914759626e+09 _runtime‚ 529.699943932ƒsÊ´Ä _runtime‚ 529.699943932 _step‚20 ' lr-AdamW/pg1‚0.00022725016506045388 ' lr-AdamW/pg2‚0.00022725016506045388 trainer/global_step‚299 % _timestamp‚1.7885212914759626e+09_òUý0µ+ _wandbruntime‚529  _runtime‚529SÜÖί'¶©O IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3249318 _step‚21 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚11803544903680 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5494989 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚8041106 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 599451682304 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5494991 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 588455144960 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 300049920 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 586786633372 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 327892480 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚11859499841772 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚13511249 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 599740971520 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚12391767147008 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2097343 % _timestamp‚1.7885212914864788e+09 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1309 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚12392018051584 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5346661 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 _runtime‚ 529.713306615 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚11803562906624 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚12522290475520 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚11272696169040 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚12392018051584 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚713 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚13122055387136 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 289289216 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3249318 _step‚21 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚11803544903680 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5494989 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚8041106 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 599451682304 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5494991 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 588455144960 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 300049920 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 586786633372 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 327892480 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚11859499841772 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚13511249 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 599740971520 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚12391767147008 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2097343 % _timestamp‚1.7885212914864788e+09 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1309 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚12392018051584 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5346661 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 _runtime‚ 529.713306615 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚11803562906624 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚12522290475520 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚11272696169040 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚12392018051584 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚713 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚13122055387136 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 289289216 IDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2103972 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚11898742675286 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚11842830940672 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚11310226526464 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 _runtime‚ 531.575546347 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚13581090 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚8042952 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚13556216 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚13581090 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚12563974445568 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚11310243565824 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚8067824 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚11842848943616 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5364092 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_poolYÂ~ë .peak‚ 6236507136 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚12433268127744 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 _step‚22 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚13556216 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 327892480 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚11842830940672 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5513264 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 601733659648 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚12433268127744 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚8067824 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5364780 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚12433017209856 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 590419184128 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚11842830940672 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚11310226526464 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 _runtime‚ 531.575546347 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚13581090 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚8042952 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚13556216 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚13581090 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚12563974445568 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚11310243565824 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚8067824 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚11842848943616 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5364092 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚12433268127744 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 _step‚22 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚13556216 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 327892480 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚11842830940672 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5513264 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 601733659648 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚12433268127744 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚8067824 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5364780 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚12433017209856 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 590419184128 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2103972 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚11898742675286 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608Z×0»+ _wandbruntime‚531  _runtime‚531v¢)ýN¼È /loss/SRqqt1/detection_loss‚4.10930061340332 epoch‚0 trainer/global_step‚299 .loss/FBt1/assignment_loss‚0.65865422072851 .loss/FRt1/detection_loss‚4.575592984755834 .loss/FRt3/detection_loss‚3.040106991827488 1loss/FBt2/assignment_loss‚0.25100836553848443 .loss/FRt4/detection_loss‚3.158805254770785 .loss/FBt2/detection_loss‚2.352525798973704 /loss/FRt3/assignment_loss‚7.293681087493897 /loss/FRt4/assignment_loss‚7.442270522214929 2loss/SRqqt3/assignment_loss‚3.3236697666228765 _step‚23 1loss/SRqqt1/assignment_loss‚3.541322626565632 0loss/SRqqt4/detection_loss‚1.628900629686005 0loss/FBt3/assignment_loss‚0.7494121640920639 0loss/FBt4/assignment_loss‚0.3489596504774509 1loss/SRqqt3/detection_loss‚2.3851807802205993 /loss/FBt3/detection_loss‚16.650629818439487 .loss/FBt4/detection_loss‚4.797932683518439 &loss/total_loss‚100.60471723601822 1loss/SRqqt2/detection_loss‚2.0621712414752533 /loss/FRt2/detection_loss‚2.5646137475967405 /loss/FRt1/assignment_loss‚7.356602231661479 /loss/FRt2/assignment_loss‚7.199013909426602 2loss/SRqqt2/assignment_loss‚3.3899398754382957 2loss/SRqqt4/assignment_loss‚3.9445737174579074 .loss/FBt1/detection_loss‚7.779848553632435 % _timestamp‚1.7885212933572588e+09 _runtime‚ 531.576667869b"íqR½Ì 0loss/SRqqt4/detection_loss‚1.628900629686005 0loss/FBt3/assignment_loss‚0.7494121640920639 0loss/FBt4/assignment_loss‚0.3489596504774509 1loss/SRqqt3/detection_loss‚2.3851807802205993 /loss/FBt3/detection_loss‚16.650629818439487 .loss/FBt4/detection_loss‚4.797932683518439 &loss/total_loss‚100.60471723601822 1loss/SRqqt2/detection_loss‚2.0621712414752533 /loss/FRt2/detection_loss‚2.5646137475967405 /loss/FRt1/assignment_loss‚7.356602231661479 /loss/FRt2/assignment_loss‚7.199013909426602 2loss/SRqqt2/assignment_loss‚3.3899398754382957 2loss/SRqqt4/assignment_loss‚3.9445737174579074 .loss/FBt1/detection_loss‚7.779848553632435 % _timestamp‚1.7885212933572588e+09 _runtime‚ 531.576667869 /loss/SRqqt1/detection_loss‚4.10930061340332 epoch‚0 trainer/global_step‚299 .loss/FBt1/assignment_loss‚0.65865422072851 .loss/FRt1/detection_loss‚4.575592984755834 .loss/FRt3/detection_loss‚3.040106991827488 1loss/FBt2/assignment_loss‚0.25100836553848443 .loss/FRt4/detection_loss‚3.158805254770785 .loss/FBt2/detection_loss‚2.352525798973704 /loss/FRt3/assignment_loss‚7.293681087493897 /loss/FRt4/assignment_loss‚7.442270522214929 2loss/SRqqt3/assignment_loss‚3.3236697666228765 _step‚23 1loss/SRqqt1/assignment_loss‚3.541322626565632-Þõ›¾‚(: ÕÖêÔ©‡éö gpu.0.gpu‚21.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚105.24' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.048000000000002 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0˜ÉÁZ¿‚(:Ï ÕÖêÔϧý÷ disk.sda2.out‚0! disk.nvme0n1.out‚ 9401.2890625 disk.sda1.in‚0 disk.dm-1.in‚0' proc.memory.availableMB‚ 560.8828125 cpu‚25.298206693442346 network.sent‚499565 network.recv‚259664 disk.drbd1260.out‚ 0.07421875 disk.sda3.out‚16.84375! disk.nvme0n1.in‚ 1285.92578125 disk.sda1.out‚0 disk.dm-1.out‚16.84375# memory_percent‚96.5766429901123) disk./.usagePercent‚78.85435087262114% disk./.usageGB‚1174.9868125915527 disk.dm-0.out‚ 9401.2890625 disk.sda.in‚0 disk.sda.out‚16.84375# disk.nvme0c0n1.in‚ 1285.92578125# proc.memory.rssMB‚ 2814.47265625) proc.memory.percent‚17.17817783355713 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.dm-0.in‚ 1285.92578125 disk.sda3.in‚0# disk.nvme0c0n1.out‚ 9401.2890625 proc.cpu.threads‚17Wß‘À‚(:† äÖêÔé¹÷ gpu.0.gpu‚23.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚105.363' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.0726 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚07„ënÁ‚(:ã äÖêÔåéŽø disk.dm-1.in‚0 disk.drbd1260.in‚0' disk.nvme0c0n1.out‚9742.53271484375$ memory_percent‚96.57533168792725 cpu‚25.29493989784489 proc.cpu.threads‚17 network.sent‚549777) disk./.usagePercent‚78.85435471273415% disk./.usageGB‚1174.9868698120117 disk.sda2.in‚0 disk.dm-0.in‚ 1285.92578125" disk.dm-0.out‚9742.53271484375 disk.sda.out‚ 17.27734375 disk.sda3.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda.in‚0 disk.sda3.out‚ 17.27734375# disk.nvme0c0n1.in‚ 1285.92578125# proc.memory.rssMB‚ 2814.47265625) proc.memory.percent‚17.17817783355713 network.recv‚266077! disk.nvme0n1.in‚ 1285.92578125% disk.nvme0n1.out‚9742.53271484375 disk.dm-1.out‚ 17.27734375 disk.drbd1260.out‚0.078125( proc.memory.availableMB‚ 561.09765625™R@ñ‘‚(:† óÖêÔ½ëªö gpu.0.gpu‚17.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚117.551' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.5102 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ž€nÂ(:ã óÖêÔ„š•ø cpu‚25.290630701910583) disk./.usagePercent‚78.85278359449653# disk.nvme0c0n1.in‚ 1285.92578125! disk.nvme0n1.in‚ 1285.92578125 disk.sda1.in‚0 disk.sda.out‚ 17.69921875# proc.memory.rssMB‚ 2814.47265625) proc.memory.percent‚17.17817783355713 proc.cpu.threads‚17 disk.sda1.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 17.69921875 disk.drbd1260.out‚0.078125" disk.dm-0.out‚9935.13427734375 network.sent‚555619 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.sda.in‚0 disk.sda3.out‚ 17.69921875$ memory_percent‚96.57530784606934' proc.memory.availableMB‚ 561.1015625 network.recv‚270470% disk./.usageGB‚1174.9634590148926' disk.nvme0c0n1.out‚9935.13427734375% disk.nvme0n1.out‚9935.13427734375 disk.sda2.out‚0 disk.dm-0.in‚ 1285.92578125 disk.sda3.in‚0†²Í!lÄ‚(:á ‚×êÔ§Ëø proc.cpu.threads‚17" disk.nvme0c0n1.in‚ 1434.1640625$ disk./.usageGB‚1175.218074798584 disk.sda2.in‚0 disk.dm-0.in‚ 1434.1640625# disk.dm-0.out‚10276.70849609375 disk.sda3.in‚0 disk.drbd1260.out‚ 0.08203125$ memory_percent‚96.57378196716309 network.recv‚273922 disk.sda2.out‚0 disk.sda.out‚ 18.1484375( disk.nvme0c0n1.out‚10276.70849609375 disk.dm-1.in‚0 disk.dm-1.out‚ 18.1484375' proc.memory.availableMB‚ 561.3515625# proc.memory.rssMB‚ 2814.47265625 network.sent‚561118) disk./.usagePercent‚78.86987107337691 disk.sda.in‚0 disk.nvme0n1.in‚ 1434.1640625 disk.sda1.in‚0 disk.drbd1260.in‚0 cpu‚25.285756008951132 disk.sda3.out‚ 18.1484375& disk.nvme0n1.out‚10276.70849609375 disk.sda1.out‚0) proc.memory.percent‚17.17817783355713ʬN‹‘Å‚(:† ‚×êÔ¦Œ¤ø gpu.0.gpu‚17.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚137.203' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.4406 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0r¾l½œÆ‚(:‘ ‘×êÔ‰¹ý÷ gpu.0.gpu‚23.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚104.059' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚20.811799999999998 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0x×.lhÇ‚(:Ý ‘×êÔ¥Ÿú proc.cpu.threads‚17$ disk./.usageGB‚1175.186912536621 disk.drbd1260.out‚ 0.08203125 disk.sda1.in‚0 disk.dm-1.in‚0& proc.memory.availableMB‚ 561.109375# proc.memory.rssMB‚ 2814.47265625) proc.memory.percent‚17.17817783355713 cpu‚25.282448137902605 disk.sda3.out‚ 18.515625 network.sent‚567157 disk.drbd1260.in‚0 disk.sda2.out‚0# disk.dm-0.out‚10750.45849609375 disk.sda.in‚0" disk.nvme0c0n1.in‚ 1434.1640625& disk.nvme0n1.out‚10750.45849609375 disk.sda2.in‚0 disk.dm-0.in‚ 1434.1640625 disk.sda.out‚ 18.515625( disk.nvme0c0n1.out‚10750.45849609375 disk.nvme0n1.in‚ 1434.1640625 disk.sda1.out‚0 disk.dm-1.out‚ 18.515625$ memory_percent‚96.57526016235352 network.recv‚278241) disk./.usagePercent‚78.86777974782972 disk.sda3.in‚0ÙOÈ‚(:…  ×êÔö·÷ gpu.0.gpu‚16.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚92.852' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.5704 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0G¼©iÉ‚(:Þ  ×êÔôˆ‚ø disk.nvme0n1.in‚ 1434.1640625% disk.nvme0n1.out‚10808.9951171875% proc.memory.availableMB‚ 561.34375 network.sent‚573785 disk.dm-1.out‚ 19.16015625 cpu‚25.279266207341045 proc.cpu.threads‚17) disk./.usagePercent‚78.88125624043965% disk./.usageGB‚1175.3877220153809 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 1434.1640625' disk.nvme0c0n1.out‚10808.9951171875# memory_percent‚96.5738296508789# proc.memory.rssMB‚ 2814.47265625) proc.memory.percent‚17.17817783355713 network.recv‚283807 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0 disk.dm-0.in‚ 1434.1640625 disk.sda.in‚0 disk.sda.out‚ 19.16015625 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.0859375 disk.sda2.in‚0" disk.dm-0.out‚10808.9951171875 disk.sda3.out‚ 19.160156254Æ0Ê+ _wandbruntime‚623  _runtime‚623O+æ#ÅË¿ $ _timestamp‚1.788521384976109e+09 _runtime‚ 623.196320675 _step‚24 ' lr-AdamW/pg1‚0.00026525186490333915 ' lr-AdamW/pg2‚0.00026525186490333915 trainer/global_step‚349='ÉÌà ' lr-AdamW/pg2‚0.00026525186490333915 trainer/global_step‚349 $ _timestamp‚1.788521384976109e+09 _runtime‚ 623.196320675 _step‚24 ' lr-AdamW/pg1‚0.00026525186490333915N40Í+ _wandbruntime‚623  _runtime‚623ô=é!ΩO _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚13767864756224 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6230511 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 300049920 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚9354212 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚14454535607808 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 699455216128 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚705 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚9379058 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 686670851584 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2441798 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚14454284703232 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1290 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚13149214040240 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚14454284703232 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1309 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 686437949952 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 319520768 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚9354212 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 684495700344 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚15787799 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 327892480 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚6408741 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚6408739 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 699750796800 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚13767846753280 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚13767846753280 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 _step‚25 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 trainer/global_step‚349 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6231218 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6231218 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚14454284703232 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1290 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚13149214040240 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚14454284703232 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1309 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 686437949952 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 319520768 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚9354212 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 684495700344 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚15787799 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 327892480 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚6408741 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚6408739 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 699750796800 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚13767846753280 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚13767846753280 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 _step‚25 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 trainer/global_step‚349 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 IDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚14495785683968 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚13807132790272 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6427014 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3800220 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚15349935037440 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 686683163074 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚663 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚13186761437024 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3800222 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚13873444600098 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚15832792 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚14648196888064 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 300049920 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6427016 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚15807918 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7885213859866102e+09 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 688401975808 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚14648172947968 trainer/global_step‚349 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚15349630210048 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6248786 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚15832792 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚15807918 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚14495534766080 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚13807132790272 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 688634890752 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6248121 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚665 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚13807150793216 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚26 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6427016{`¯½&Ò·M FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚13807132790272 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 688634890752 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6248121 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚665 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚13807150793216 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚26 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6427016 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚9405776 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚9380904 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1290 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚14495785683968 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚13807132790272 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6427014 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3800220 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚15349935037440 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 686683163074 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚663 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚13186761437024 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3800222 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚13873444600098 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚15832792 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚14648196888064 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 300049920 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6427016 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚15807918 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7885213859866102e+09 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 688401975808 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚14648172947968 trainer/global_step‚349 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚15349630210048 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6248786 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚15832792 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚15807918 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚14495534766080é-§0Ó+ _wandbruntime‚624  _runtime‚624}Ì ªÔÇ /loss/FRt4/assignment_loss‚7.383563015196058 0loss/SRqqt1/detection_loss‚5.473083233391797 -loss/FBt4/detection_loss‚3.47516036380169 2loss/SRqqt2/assignment_loss‚3.1460942830124945 1loss/SRqqt4/assignment_loss‚3.718403140703837 0loss/FBt2/assignment_loss‚0.5263375668916578 1loss/FBt4/assignment_loss‚0.14452989658863327 0loss/SRqqt4/detection_loss‚4.389170626260374 .loss/FBt2/detection_loss‚3.150082867085403 _step‚27 0loss/FBt1/assignment_loss‚0.5826617242472077 0loss/FBt3/assignment_loss‚0.8311716318130493 .loss/FRt1/detection_loss‚5.148092869669199 .loss/FBt1/detection_loss‚4.172159697070266 .loss/FRt3/detection_loss‚2.535449500989031 .loss/FRt4/detection_loss‚4.122144655221039 _runtime‚ 624.209659816 /loss/FRt2/assignment_loss‚7.226440302530924 trainer/global_step‚349 &loss/total_loss‚137.49605225556815 /loss/FRt3/assignment_loss‚¥?£7.422143088446723 1loss/SRqqt2/detection_loss‚1.3478140234560483 1loss/SRqqt3/detection_loss‚2.0317439287900925 -loss/FBt3/detection_loss‚54.2436797618866 epoch‚0 % _timestamp‚1.7885213859901462e+09 /loss/FRt2/detection_loss‚2.4597606632858513 /loss/FRt1/assignment_loss‚7.215608060359955 2loss/SRqqt1/assignment_loss‚3.4666621066905834 1loss/SRqqt3/assignment_loss‚3.284095248179649í¯–¡QÕË 2loss/SRqqt1/assignment_loss‚3.4666621066905834 1loss/SRqqt3/assignment_loss‚3.284095248179649 /loss/FRt4/assignment_loss‚7.383563015196058 0loss/SRqqt1/detection_loss‚5.473083233391797 -loss/FBt4/detection_loss‚3.47516036380169 2loss/SRqqt2/assignment_loss‚3.1460942830124945 1loss/SRqqt4/assignment_loss‚3.718403140703837 0loss/FBt2/assignment_loss‚0.5263375668916578 1loss/FBt4/assignment_loss‚0.14452989658863327 0loss/SRqqt4/detection_loss‚4.389170626260374 .loss/FBt2/detection_loss‚3.150082867085403 _step‚27 0loss/FBt1/assignment_loss‚0.5826617242472077 0loss/FBt3/assignment_loss‚0.8311716318130493 .loss/FRt1/detection_loss‚5.148092869669199 .loss/FBt1/detection_loss‚4.172159697070266 .loss/FRt3/detection_loss‚2.535449500989031 .loss/FRt4/detection_loss‚4.122144655221039 _runtime‚ 624.209659816 /loss/FRt2/assignment_loss‚7.226440302530924 trainer/global_step‚349 &loss/total_loss‚137.49605225556815 /loss/FRt3/assignment_loss‚7.422143088446723 1loss/SRqqt2/detection_loss‚1.3478140234560483 1loss/SRqqt3/detection_loss‚2.0317439287900925 -loss/FBt3/detection_loss‚54.2436797618866 epoch‚0 % _timestamp‚1.7885213859901462e+09 /loss/FRt2/detection_loss‚2.4597606632858513 /loss/FRt1/assignment_loss‚7.215608060359955º·T*‘Ö‚(:† ¯×êÔ€âÂø gpu.0.gpu‚19.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚101.848' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.3696 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µ–a·jׂ(:ß ¯×êÔþãÊú disk.sda2.in‚0 disk.sda3.out‚ 19.4921875 disk.sda1.out‚0 disk.sda.in‚0 disk.sda.out‚ 19.4921875 disk.sda3.in‚0$ memory_percent‚96.57442569732666# proc.memory.rssMB‚ 2814.47265625) proc.memory.percent‚17.17817783355713 proc.cpu.threads‚17 disk.drbd1260.in‚0) disk./.usagePercent‚78.88121937535473% disk./.usageGB‚1175.3871726989746 disk.nvme0n1.in‚ 1434.1640625" disk.dm-0.out‚11152.7958984375( proc.memory.availableMB‚ 561.24609375 cpu‚25.275487114953826" disk.nvme0c0n1.in‚ 1434.1640625' disk.nvme0c0n1.out‚11152.7958984375 disk.drbd1260.out‚ 0.0859375 disk.sda2.out‚0 disk.dm-0.in‚ 1434.1640625 disk.dm-1.out‚ 19.4921875 network.sent‚579095 network.recv‚287351% disk.nvme0n1.out‚11152.7958984375 disk.sda1.in‚0 disk.dm-1.in‚0òR¶œØ‚(:‘ ¾×êÔîß¼ö gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚125.733' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.146600000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0­>F=iÙ‚(:Þ ¾×êÔ¤Þø disk.sda2.out‚0 disk.sda.out‚ 19.80859375 disk.nvme0c0n1.in‚ 1486.84375 disk.dm-1.out‚ 19.80859375 disk.drbd1260.out‚ 0.08984375) proc.memory.percent‚17.17817783355713 network.sent‚628145 disk.sda2.in‚0# disk.dm-0.out‚11403.65576171875 disk.sda.in‚0 disk.nvme0n1.in‚ 1486.84375& disk.nvme0n1.out‚11403.65576171875 disk.dm-1.in‚0& proc.memory.availableMB‚ 561.203125) disk./.usagePercent‚78.88121963136226% disk./.usageGB‚1175.3871765136719 disk.dm-0.in‚ 1486.84375 disk.sda3.in‚0 disk.sda3.out‚ 19.80859375 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚96.57468795776367 network.recv‚292399( disk.nvme0c0n1.out‚11403.65576171875 disk.drbd1260.in‚0# proc.memory.rssMB‚ 2814.47265625 cpu‚25.272563910673835 proc.cpu.threads‚17îÓ2ÄÚ‚(:… Í×êÔÐó¶÷ gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚102.422' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.4844 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ã’ìŠrÛ‚(:ç Í×êÔæÀ¹û) disk./.usagePercent‚78.88121835132459 disk.drbd1260.out‚ 0.08984375 disk.sda.in‚0 disk.sda3.in‚0( disk.nvme0c0n1.out‚11768.80419921875$ memory_percent‚96.57464027404785 network.recv‚297067 disk.sda1.out‚0 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 1551.70703125 disk.sda.out‚ 20.23046875 disk.sda3.out‚ 20.23046875 disk.sda1.in‚0 disk.dm-1.out‚ 20.23046875 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 1551.70703125& disk.nvme0n1.out‚11768.80419921875' proc.memory.availableMB‚ 561.2109375) proc.memory.percent‚17.17817783355713 cpu‚25.26907655034206# disk.dm-0.out‚11768.80419921875 network.sent‚635981% disk./.usageGB‚1175.3871574401855! disk.nvme0n1.in‚ 1551.70703125# proc.memory.rssMB‚ 2814.47265625 proc.cpu.threads‚17kDö)Ü‚(:… Ü×êÔúá¶ø gpu.0.gpu‚19.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚103.195' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.639 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Yë:hÝ‚(:Ý Ü×êÔ´ÒŠü disk.sda1.out‚0 disk.sda2.out‚0 disk.dm-0.in‚ 1551.70703125 disk.sda.in‚0 disk.sda.out‚ 20.7265625) proc.memory.percent‚17.17817783355713$ disk.nvme0n1.out‚12019.513671875 disk.dm-1.out‚ 20.7265625 disk.drbd1260.out‚0.09375 disk.sda3.out‚ 20.7265625$ memory_percent‚96.57490253448486 proc.cpu.threads‚17% disk./.usageGB‚1175.3636894226074# disk.nvme0c0n1.in‚ 1551.70703125! disk.nvme0n1.in‚ 1551.70703125 disk.sda1.in‚0 disk.drbd1260.in‚0 disk.sda2.in‚0( proc.memory.availableMB‚ 561.16796875# proc.memory.rssMB‚ 2814.47265625 network.sent‚640327 network.recv‚301649 disk.dm-1.in‚0! disk.dm-0.out‚12019.513671875 disk.sda3.in‚0 cpu‚25.266224942254393) disk./.usagePercent‚78.87964339297395& disk.nvme0c0n1.out‚12019.513671875çŒÜÒÞ‚(:… ë×êÔ·ÛÓö gpu.0.gpu‚7.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚115.147' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.0294 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ØYËcß‚(:Ø ë×êÔ㨆ú disk.sda1.out‚0 disk.dm-1.in‚0 disk.drbd1260.out‚0.09375% disk./.usageGB‚1175.3871765136719# disk.nvme0c0n1.in‚ 1551.70703125 disk.sda2.in‚0 disk.sda.out‚ 21.1328125# memory_percent‚96.5764045715332& proc.memory.availableMB‚ 560.921875) proc.memory.percent‚17.17817783355713& disk.nvme0c0n1.out‚12242.544921875 disk.dm-1.out‚ 21.1328125 disk.sda2.out‚0 disk.sda3.out‚ 21.1328125 cpu‚25.2630069212011 disk.sda1.in‚0 disk.drbd1260.in‚0 disk.dm-0.in‚ 1551.70703125! disk.dm-0.out‚12242.544921875 disk.sda.in‚0 disk.sda3.in‚0# proc.memory.rssMB‚ 2814.47265625 proc.cpu.threads‚17 network.sent‚645455 network.recv‚305245) disk./.usagePercent‚78.88121963136226! disk.nvme0n1.in‚ 1551.70703125$ disk.nvme0n1.out‚12242.544921875Z­Œà‚(:„ ú×êÔòŸôö gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚29.0 gpu.0.powerWatts‚92.852' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.5704 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0äç1Ërá‚(:ç ú×êÔÈÒ¬ø! disk.nvme0n1.in‚ 1551.70703125& disk.nvme0n1.out‚12433.20751953125 disk.sda1.in‚0 network.recv‚308684 disk.sda3.out‚ 21.53515625( disk.nvme0c0n1.out‚12433.20751953125 disk.dm-1.out‚ 21.53515625) proc.memory.percent‚17.17817783355713 disk.sda3.in‚0% disk./.usageGB‚1175.3993034362793 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.09765625# disk.dm-0.out‚12433.20751953125 disk.sda1.out‚0$ memory_percent‚96.57645225524902' proc.memory.availableMB‚ 560.9140625 disk.sda2.in‚0 disk.sda.in‚0# disk.nvme0c0n1.in‚ 1551.70703125 disk.dm-1.in‚0# proc.memory.rssMB‚ 2814.47265625 cpu‚25.260550871199726 proc.cpu.threads‚17 disk.sda.out‚ 21.53515625 network.sent‚650594( disk./.usagePercent‚78.8820334793136 disk.sda2.out‚0 disk.dm-0.in‚ 1551.70703125°²Ù^0â+ _wandbruntime‚718  _runtime‚718­ýkØÄã¾ & lr-AdamW/pg2‚0.0003032535647462244 trainer/global_step‚399 % _timestamp‚1.7885214802420568e+09 _runtime‚ 718.462703498 _step‚28 & lr-AdamW/pg1‚0.0003032535647462244-ßÈä & lr-AdamW/pg1‚0.0003032535647462244 & lr-AdamW/pg2‚0.0003032535647462244 trainer/global_step‚399 % _timestamp‚1.7885214802420568e+09 _runtime‚ 718.462703498 _step‚28Ê„Ç›0å+ _wandbruntime‚718  _runtime‚718—^Ux³'æ­O ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 784886558208 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 782662714388 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 784886558208 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 _step‚29 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚16517053164032 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚16690711420416 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚15025748950800 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 % _timestamp‚1.7885214802454362e+09 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚15732166605824 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7322489 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚16690687480320 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚15732166605824 trainer/global_step‚399 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4329520 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7118390 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 799576728576 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚687 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚17490264208896 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2788185 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚15732148602880 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7322491 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7322491 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1309 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 327892480 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 784653656576 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚10692164 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚15808165652292 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 782433740852 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚16516802259456 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚15025731911440 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚16516802259456 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2788870 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚685 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚15732166605824 trainer/global_step‚399 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4329520 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7118390 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 799576728576 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚687 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚17490264208896 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2788185 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚15732148602880 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7322491 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7322491 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1309 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 327892480 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 784653656576 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚10692164 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚15808165652292 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 782433740852 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚16516802259456 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚15025731911440X7½â‘è‚(:† ‰ØêÔä¡â÷ gpu.0.gpu‚22.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚101.967' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.3934 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ú ’–ré‚(:ç ‰ØêÔìÈù disk.sda3.out‚ 21.83984375# disk.nvme0c0n1.in‚ 1683.41015625 disk.sda1.in‚0 disk.dm-1.in‚0) disk./.usagePercent‚78.88202912718553 disk.sda2.out‚0 disk.sda.out‚ 21.83984375 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.09765625 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda3.in‚0! disk.nvme0n1.in‚ 1683.41015625& disk.nvme0n1.out‚12594.37548828125' proc.memory.availableMB‚ 561.4453125# proc.memory.rssMB‚ 2814.47265625) proc.memory.percent‚17.17817783355713 network.recv‚313478 disk.dm-0.in‚ 1683.41015625( disk.nvme0c0n1.out‚12594.37548828125 disk.dm-1.out‚ 21.83984375$ memory_percent‚96.57320976257324 cpu‚25.25799422646257 network.sent‚656655% disk./.usageGB‚1175.3992385864258# disk.dm-0.out‚12594.37548828125 disk.sda.in‚0¤n 0ê+ _wandbruntime‚720  _runtime‚720g»•íë¶M FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚15771434639872 FDeviceStatsMonitor.on_train_batch_end/requested_bytex´Ï"s.all.peak‚ 6741051909 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7135992 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 327892480 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 300049920 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚15063279308224 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚689 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4340322 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚16558303240192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚18059620 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2795670 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚10718856 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 302730240 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 786850597376 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚17534250792448 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 _step‚30 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚10718856 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚15771434639872 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 786617682432 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚18084494 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚18059620 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7340764 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚687 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 278790144 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚10743728 _runtime‚ 720.835173599 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 786617682432 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚16558052322304 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚10743728 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚15847654498702 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7135303 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4340320 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 $ _timestamp‚1.788521482611562e+09 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 801855401984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7340766 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2794983 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1290 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚16732395390464 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 784392229838 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 786850597376 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7340766 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚16732371450368 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚16558303240192 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 784621203582±ÔÆvÀ&ìºM >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1290 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚16732395390464 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 784392229838 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 786850597376 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7340766 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚16732371450368 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚16558303240192 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 784621203582 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚15771434639872 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7135992 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 327892480 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 300049920 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚15063279308224 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚689 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4340322 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚16558303240192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚18059620 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2795670 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚10718856 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 302730240 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 786850597376 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚17534250792448 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 _step‚30 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚10718856 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚15771434639872 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 786617682432 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚18084494 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚18059620 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7340764 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚687 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 278790144 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚10743728 _runtime‚ 720.835173599 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 786617682432 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚16558052322304 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚10743728 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚15847654498702 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7135303 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4340320 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 $ _timestamp‚1.788521482611562e+09 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 801855401984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7340766 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2794983ÞXå0í+ _wandbruntime‚720  _runtime‚720(£ÝðQîË /loss/SRqqt1/detection_loss‚4.38021358270799 1loss/SRqqt3/detection_loss‚2.8550668694795323 % _timestamp‚1.7885214826187558e+09 /loss/FRt3/assignment_loss‚7.309959462710789 1loss/SRqqt1/assignment_loss‚3.674525245543449 /loss/FRt4/detection_loss‚3.8621744730988063 1loss/SRqqt2/detection_loss‚1.4713258219077556 _runtime‚ 720.839573208 2loss/SRqqt2/assignment_loss‚3.1727318959693385 0loss/FBt1/assignment_loss‚0.4897563214712889 1loss/FBt2/assignment_loss‚0.34790692658715694 /loss/FRt3/detection_loss‚2.5372175936188017 /loss/FBt1/detection_loss‚4.7121557685835604 /loss/FBt4/detection_loss‚3.6078890498195375 trainer/global_step‚399 _step‚31 .loss/FRt1/assignment_loss‚7.19509294629097 1loss/FBt4/assignment_loss‚0.27350474056410823 0loss/SRqqt4/detection_loss‚4.907278377901424 0loss/FBt3/assignment_loss‚0.9170830141414296 -loss/FBt3/detection_loss‚9.01245310089805 $loss/total_loss‚91.0430126667946 epoch‚0 /loss/FRt1/detection_loss‚4.0580781027674675 0loss/FRt2/assignment_loss‚7.2138590812683105 2loss/SRqqt4/assignment_loss‚3.7556600137190386 /loss/FRt2/detection_loss‚2.1647912621025056 /loss/FRt4/assignment_loss‚7.443610526419975 0loss/SRqqt3/assignment_loss‚3.30436902321302 /loss/FBt2/detection_loss‚2.3763094660102757—Þ|ßUïÏ 2loss/SRqqt2/assignment_loss‚3.1727318959693385 0loss/FBt1/assignment_loss‚0.4897563214712889 1loss/FBt2/assignment_loss‚0.34790692658715694 /loss/FRt3/detection_loss‚2.5372175936188017 /loss/FBt1/detection_loss‚4.7121557685835604 /loss/FBt4/detection_loss‚3.6078890498195375 trainer/global_step‚399 _step‚31 .loss/FRt1/assignment_loss‚7.19509294629097 1loss/FBt4/assignment_loss‚0.27350474056410823 0loss/SRqqt4/detection_loss‚4.907278377901424 0loss/FBt3/assignment_loss‚0.9170830141414296 -loss/FBt3/detection_loss‚9.01245310089805 $loss/total_loss‚91.0430126667946 epoch‚0 /loss/FRt1/detection_loss‚4.0580781027674675 0loss/FRt2/assignment_loss‚7.2138590812683105 2loss/SRqqt4/assignment_loss‚3.7556600137190386 /loss/FRt2/detection_loss‚2.1647912621025056 /loss/FRt4/assignment_loss‚7.443610526419975 0loss/SRqqt3/assignment_loss‚3.30436902321302 /loss/FBt2/detection_loss‚2.3763094660102757 /loss/SRqqt1/detection_loss‚4.38021358270799 1loss/SRqqt3/detection_loss‚2.8550668694795323 % _timestamp‚1.7885214826187558e+09 /loss/FRt3/assignment_loss‚7.309959462710789 1loss/SRqqt1/assignment_loss‚3.674525245543449 /loss/FRt4/detection_loss‚3.8621744730988063 1loss/SRqqt2/detection_loss‚1.4713258219077556 _runtime‚ 720.839573208Vš“œœð‚(:‘ ˜ØêÔêë¯ö gpu.0.gpu‚19.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚105.603' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.120599999999996 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0E9‚ßlñ‚(:á ˜ØêÔ¸Óñ÷# disk.dm-0.out‚12715.18310546875 disk.sda.in‚0# disk.nvme0c0n1.in‚ 1683.41015625& disk.nvme0n1.out‚12715.18310546875# proc.memory.rssMB‚ 2814.47265625 cpu‚25.25540958650704! disk.nvme0n1.in‚ 1683.41015625% disk./.usageGB‚1175.3990859985352 disk.sda.out‚ 22.2421875 disk.sda3.out‚ 22.2421875( disk.nvme0c0n1.out‚12715.18310546875 disk.sda1.in‚0 disk.drbd1260.out‚ 0.1015625 disk.sda2.out‚0 disk.sda3.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 22.2421875 disk.sda2.in‚0% proc.memory.availableMB‚ 561.21875) proc.memory.percent‚17.17817783355713 proc.cpu.threads‚17 network.sent‚695536 network.recv‚319188) disk./.usagePercent‚78.88201888688415 disk.dm-0.in‚ 1683.41015625 disk.sda1.out‚0 disk.drbd1260.in‚0$ memory_percent‚96.57459259033203£NEºjò‚(:ß §ØêԇĜø) disk./.usagePercent‚78.88202016692183% disk./.usageGB‚1175.3991050720215 disk.sda2.out‚0 disk.dm-0.in‚ 1683.41015625! disk.dm-0.out‚13128.587890625 disk.sda.in‚0 disk.sda.out‚ 22.7578125 network.recv‚323300! disk.nvme0n1.in‚ 1683.41015625$ disk.nvme0n1.out‚13128.587890625 disk.drbd1260.out‚ 0.1015625 disk.sda2.in‚0 disk.sda3.in‚0$ memory_percent‚96.57618999481201# proc.memory.rssMB‚ 2814.47265625# disk.nvme0c0n1.in‚ 1683.41015625& disk.nvme0c0n1.out‚13128.587890625 disk.sda1.in‚0 disk.dm-1.out‚ 22.7578125 disk.sda3.out‚ 22.7578125( proc.memory.availableMB‚ 560.95703125) proc.memory.percent‚17.17817783355713 cpu‚25.252790688466174 disk.sda1.out‚0 disk.dm-1.in‚0 disk.drbd1260.in‚0 proc.cpu.threads‚17 network.sent‚732715xU ‘ó‚(:† §ØêÔ…¨âú gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚139.933' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.9866 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ì%<¹ô‚(:„ ¶ØêÔÁÊÞ÷ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚29.0 gpu.0.powerWatts‚92.744' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.5488 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0›>ß$hõ‚(:Ý ¶ØêÔÕ„ù) proc.memory.percent‚17.17817783355713 network.recv‚327115" disk.dm-0.out‚13343.7841796875' disk.nvme0c0n1.out‚13343.7841796875 cpu‚25.250544258001373 network.sent‚737915 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0 disk.dm-0.in‚ 1683.41015625! disk.nvme0n1.in‚ 1683.41015625$ memory_percent‚96.57621383666992 proc.cpu.threads‚17( disk./.usagePercent‚78.8820206789369 disk.dm-1.out‚ 23.1796875 disk.drbd1260.in‚0 disk.drbd1260.out‚0.109375 disk.sda2.in‚0 disk.sda.in‚0# disk.nvme0c0n1.in‚ 1683.41015625% disk.nvme0n1.out‚13343.7841796875$ disk./.usageGB‚1175.399112701416 disk.sda2.out‚0 disk.sda.out‚ 23.1796875 disk.sda3.in‚0 disk.sda3.out‚ 23.1796875& proc.memory.availableMB‚ 560.953125# proc.memory.rssMB‚ 2814.47265625;ŠHkö‚(:„ ÅØêÔ‘â§ø gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚92.852' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.5704 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ó†õa÷‚(:Ö ÅØêÔ¶þïù disk.sda1.out‚0 disk.dm-1.out‚ 23.5703125$ memory_percent‚96.57618999481201# proc.memory.rssMB‚ 2814.47265625 proc.cpu.threads‚17 network.sent‚743748 network.recv‚331422 disk.sda.out‚ 23.5703125' disk.nvme0c0n1.out‚13696.8701171875% disk.nvme0n1.out‚13696.8701171875 disk.sda1.in‚0 disk.drbd1260.in‚0) proc.memory.percent‚17.17817783355713( disk./.usagePercent‚78.8820206789369 disk.dm-0.in‚ 1812.15625" disk.dm-0.out‚13696.8701171875 disk.sda3.out‚ 23.5703125 disk.nvme0c0n1.in‚ 1812.15625 disk.dm-1.in‚0 disk.drbd1260.out‚0.109375( proc.memory.availableMB‚ 560.95703125$ disk./.usageGB‚1175.399112701416 disk.sda2.in‚0 disk.sda.in‚0 disk.nvme0n1.in‚ 1812.15625 cpu‚25.248073221350676 disk.sda2.out‚0 disk.sda3.in‚0ƒ Hžø‚(:… ÔØêÔ”íõ gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚111.265' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.253 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆ»*uhù‚(:Ý ÔØêÔ‚ÎËù disk.nvme0c0n1.in‚ 1812.15625( disk.nvme0c0n1.out‚13893.90185546875 network.sent‚749957 disk.dm-1.in‚0 disk.dm-1.out‚ 23.99609375# disk.dm-0.out‚13893.90185546875 disk.sda.out‚ 23.99609375 disk.sda3.out‚ 23.99609375 cpu‚25.2457637431622 disk.sda2.out‚0& disk.nvme0n1.out‚13893.90185546875 disk.drbd1260.out‚ 0.11328125 disk.dm-0.in‚ 1812.15625 disk.sda.in‚0 disk.sda3.in‚0 network.recv‚336270 disk.drbd1260.in‚0$ memory_percent‚96.57492637634277' proc.memory.availableMB‚ 561.1640625# proc.memory.rssMB‚ 2814.47265625) proc.memory.percent‚17.17817783355713 proc.cpu.threads‚17) disk./.usagePercent‚78.89879429258032% disk./.usageGB‚1175.6490516662598 disk.nvme0n1.in‚ 1812.15625 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0Âú"Äú‚(:… ãØêÔ–í–ö gpu.0.gpu‚10.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚99.967' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.9934 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÒË‹jû‚(:ß ãØêÔúÂø disk.sda2.out‚0 disk.dm-0.in‚ 1812.15625# disk.dm-0.out‚14042.70263671875 disk.sda.in‚0 disk.sda3.in‚0 disk.dm-1.out‚ 24.39453125 disk.drbd1260.out‚ 0.11328125 network.sent‚755096 network.recv‚340123 disk.sda1.out‚0 disk.sda3.out‚ 24.39453125$ memory_percent‚96.57647609710693( proc.memory.availableMB‚ 560.91015625 disk.dm-1.in‚0# proc.memory.rssMB‚ 2814.47265625) proc.memory.percent‚17.17817783355713 proc.cpu.threads‚17) disk./.usagePercent‚78.89879429258032% disk./.usageGB‚1175.6490516662598 disk.nvme0n1.in‚ 1812.15625& disk.nvme0n1.out‚14042.70263671875 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda.out‚ 24.39453125 cpu‚25.24447320852627 disk.nvme0c0n1.in‚ 1812.15625( disk.nvme0c0n1.out‚14042.70263671875 disk.drbd1260.in‚0zœgÎ0ü+ _wandbruntime‚811  _runtime‚811|~zqÃý½ & lr-AdamW/pg2‚0.0003412552645891097 trainer/global_step‚449 $ _timestamp‚1.788521573091939e+09 _runtime‚ 811.311973882 _step‚32 & lr-AdamW/pg1‚0.0003412552645891097ÊpáÇþÁ $ _timestamp‚1.788521573091939e+09 _runtime‚ 811.311973882 _step‚32 & lr-AdamW/pg1‚0.0003412552645891097 & lr-AdamW/pg2‚0.0003412552645891097 trainer/global_step‚449 Õìu0ÿ+ _wandbruntime‚811  _runtime‚811ZZT; €­O N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚17696450452480 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚19674779579904 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 296452096 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 880600754896 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 trainer/global_step‚449 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8236239 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚18579319815680 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 883102264832 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1290 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚18579570720256 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8236241 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 883102264832 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚17696468455424 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8005974 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1309 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4869618 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3137060 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 899869657088 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 OCDeviceStatsMonitor.on_train_batch_start/allocati,¢-!xon.large_pool.freed‚8236239 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚20266355 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8236241 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚12054962 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚706 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚18774885982720 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3136356 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 % _timestamp‚1.7885215730941963e+09 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚17696468455424 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 327892480 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8006680 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚17696450452480 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚704 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _step‚33 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 882869363200 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4869620 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 880371781360 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 899597145088 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚16902249782640 _runtime‚ 811.314312401 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚18579319815680 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚16902266822000 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12030116 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚18774909922816 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 882869363200 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 272512000 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚17696450452480 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚704 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _step‚33 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 882869363200 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4869620 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 880371781360 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 899597145088 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚16902249782640 _runtime‚ 811.314312401 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚18579319815680 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚16902266822000 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12030116 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚18774909922816 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 882869363200 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 272512000 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚18579319815680 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 883102264832 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1290 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚18579570720256 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8236241!º€æ0‚+ _wandbruntime‚812  _runtime‚812Ž°åϽ&ƒ·M R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚18620569878528 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 884833389056 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚16939797179424 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚17735736489472 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 % _timestamp‚1.7885215740237837e+09 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 884833389056 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 327892480 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚20336196 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 285081600 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚17822356423514 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 882559244090 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚ 812.243789014 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚18816593892864 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚17822110410410 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 882330270346 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚19718468392960 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 300049920 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚17735754492416 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚20311322 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12081680 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚18816569952768 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚20311322 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8023995 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 901874500096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3143575 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 885066304000 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚34 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚18620820796416 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1309 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8254514 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚686²Å Á„»M SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚ 812.243789014 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚18816593892864 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚17822110410410 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 882330270346 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚19718468392960 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 300049920 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚17735754492416 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚20311322 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12081680 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚18816569952768 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚20311322 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8023995 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 901874500096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3143575 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 885066304000 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚34 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚18620820796416 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1309 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8254514 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚686 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚18620569878528 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 884833389056 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 >8DeviceStatsMonitor.on_train_batch_¶Ùend/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚16939797179424 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚17735736489472 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 % _timestamp‚1.7885215740237837e+09 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 884833389056 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 327892480 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚20336196 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 285081600 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚17822356423514 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 882559244090 ODeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚676 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚21858765867520 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚20859108425216 trainer/global_step‚499 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8895184 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1309 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚19660752302080 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚19660770305024 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8894508 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5409718 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 981317971456 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 300646400 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚18778767653840 _runtime‚ 867.567206143 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚13392914 _step‚37 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1290 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚22542905 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 276706304 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚22518057 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚20641837371904 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚20642088276480 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚19660770305024 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚13368068 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5409720 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚19757077475708 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9149989 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚19757323488604 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚674 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚20859084485120 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 999681382400 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 981085069824 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9149989 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚20642088276480 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚676 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚21858765867520 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚20859108425216 trainer/global_step‚499 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8895184 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1309 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚19660752302080 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚19660770305024 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8894508 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5409718 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 981317971456 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 300646400 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚18778767653840 _runtime‚ 867.567206143 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚13392914 _step‚37 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1290 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚22542905 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 276706304 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚22518057 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚20641837371904 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚20642088276480 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚19660770305024 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚13368068 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5409720 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚19757077475708 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9149989 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚19757323488604 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚674 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚20859084485120 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 999681382400 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 981085069824 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309%;ìX0”+ _wandbruntime‚868  _runtime‚868U:ƒÿ9•¸M =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚22563024 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282984448 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚22587898 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 327892480 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚18816315050624 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚20683338352640 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306924544 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 _runtime‚ 868.823798618 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9168264 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 983282010624 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5420520 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚20900768455168 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚19796566322118 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9168264 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚13419632 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚13394760 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3491300 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚19796812335222 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚22587898 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚13394760 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚19700038339072 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚20683087434752 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚22563024 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1290 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8911820 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 980497284598 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 983049095680 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9168266 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚20900792395264 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 983282010624 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚19700056342016 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚21902446378496 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1001960907776 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 980268310854 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8912476 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚13419632 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3491954 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚656 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1309[>þcÂ&–¼M FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8911820 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 980497284598 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 983049095680 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9168266 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚20900792395264 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 983282010624 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚19700056342016 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚21902446378496 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1001960907776 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 980268310854 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8912476 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚13419632 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3491954 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚656 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1309 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚22563024 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282984448 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚22587898 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 327892480 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚18816315050624 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚20683338352640 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306924544 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 _runtime‚ 868.823798618 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9168264 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 983282010624 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5420520 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚20900768455168 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚19796566322118 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9168264 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚13419632 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚13394760 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3491300 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚19796812335222 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚22587898 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚13394760 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚19700038339072 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚20683087434752 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚22563024 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1290 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751&”ñŒã0—+ _wandbruntime‚868  _runtime‚868—Mÿ®N˜È /loss/FRt1/assignment_loss‚6.857596947596623 /loss/FBt1/assignment_loss‚0.539500232636783 /loss/SRqqt1/detection_loss‚5.28270900911755 .loss/FBt1/detection_loss‚7.053607136011124 0loss/FBt4/assignment_loss‚0.5184510257514332 /loss/FBt3/detection_loss‚14.078208496173223 %loss/total_loss‚95.12597823197882 -loss/FRt3/assignment_loss‚6.2662746495214 /loss/FRt4/assignment_loss‚7.285058689117432 2loss/SRqqt1/assignment_loss‚2.9811136457655163 /loss/FRt2/detection_loss‚2.1294270473368027 /loss/FRt3/detection_loss‚2.5273888763682595 .loss/FRt4/detection_loss‚4.357763022184372 1loss/SRqqt2/detection_loss‚0.6584800842515451 epoch‚0 /loss/FRt1/detection_loss‚2.8151467855160055 0loss/SRqqt3/detection_loss‚1.854864135191038 /loss/FBt2/detection_loss‚1.8838797516408914 $ _timestamp‚1.788521630605523e+09 _runtime‚ 868.825367837 _step‚39 .loss/FBt4/detection_loss‚6.866251575095313 2loss/SRqqt3/assignment_loss‚2.6536266803741455 1loss/SRqqt4/assignment_loss‚3.655517413698394 1loss/FBt2/assignment_loss‚0.43562550392765004 /loss/FRt2/assignment_loss‚6.343027058769675 2loss/SRqqt2/assignment_loss‚3.1944865275578325 0loss/FBt3/assignment_loss‚0.8128271974953046 1loss/SRqqt4/detection_loss‚4.0751467408805055 trainer/global_step‚499!õb‰R™Ì -loss/FRt3/assignment_loss‚6.2662746495214 /loss/FRt4/assignment_loss‚7.285058689117432 2loss/SRqqt1/assignment_loss‚2.9811136457655163 /loss/FRt2/detection_loss‚2.1294270473368027 /loss/FRt3/detection_loss‚2.5273888763682595 .loss/FRt4/detection_loss‚4.357763022184372 1loss/SRqqt2/detection_loss‚0.6584800842515451 epoch‚0 /loss/FRt1/detection_loss‚2.8151467855160055 0loss/SRqqt3/detection_loss‚1.854864135191038 /loss/FBt2/detection_loss‚1.8838797516408914 $ _timestamp‚1.788521630605523e+09 _runtime‚ 868.825367837 _step‚39 .loss/FBt4/detection_loss‚6.866251575095313 2loss/SRqqt3/assignment_loss‚2.6536266803741455 1loss/SRqqt4/assignment_loss‚3.655517413698394 1loss/FBt2/assignment_loss‚0.43562550392765004 /loss/FRt2/assignment_loss‚6.343027058769675 2loss/SRqqt2/assignment_loss‚3.1944865275578325 0loss/FBt3/assignment_loss‚0.8128271974953046 1loss/SRqqt4/detection_loss‚4.0751467408805055 trainer/global_step‚499 /loss/FRt1/assignment_loss‚6.857596947596623 /loss/FBt1/assignment_loss‚0.539500232636783 /loss/SRqqt1/detection_loss‚5.28270900911755 .loss/FBt1/detection_loss‚7.053607136011124 0loss/FBt4/assignment_loss‚0.5184510257514332 /loss/FBt3/detection_loss‚14.078208496173223 %loss/total_loss‚95.12597823197882'ÍSZœš‚(:‘ ŸÙêÔç´Ôö gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚109.885' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.977000000000004 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0i¸qCn›‚(:ã ŸÙêÔŒ˜Ÿù disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda.out‚ 25.94921875 disk.sda3.out‚ 25.94921875% disk.nvme0n1.out‚15494.8369140625 disk.sda1.in‚0 cpu‚25.233913825794907 proc.cpu.threads‚17 disk.dm-0.in‚ 1964.80859375 network.sent‚820920) disk./.usagePercent‚78.83658036563939% disk./.usageGB‚1174.7220191955566 disk.dm-1.in‚0 disk.dm-1.out‚ 25.94921875 disk.drbd1260.out‚ 0.12109375# disk.nvme0c0n1.in‚ 1964.80859375 disk.sda2.in‚0" disk.dm-0.out‚15494.8369140625 disk.sda3.in‚0' disk.nvme0c0n1.out‚15494.8369140625 disk.sda1.out‚0$ memory_percent‚96.58184051513672% proc.memory.availableMB‚ 560.03125" proc.memory.rssMB‚ 2814.5546875 network.recv‚357535 disk.sda2.out‚0! disk.nvme0n1.in‚ 1964.80859375* proc.memory.percent‚17.178678512573242’pNžœ‚(:… ®ÙêÔ•ö gpu.0.gpu‚35.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚102.015' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.403 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0l®Ðúk‚(:à ®ÙêÔŒþâø% disk./.usageGB‚1174.7219848632812! disk.nvme0n1.in‚ 1964.80859375 disk.sda1.in‚0 disk.drbd1260.in‚0 disk.sda2.out‚0 disk.sda3.in‚0& proc.memory.availableMB‚ 560.765625 network.recv‚362082) disk./.usagePercent‚78.83657806157157# disk.nvme0c0n1.in‚ 1964.80859375 disk.sda2.in‚0 disk.sda.in‚0 proc.cpu.threads‚17 network.sent‚871562 disk.sda1.out‚0 disk.dm-1.out‚ 26.30078125 disk.drbd1260.out‚0.125# disk.dm-0.out‚15543.55908203125" proc.memory.rssMB‚ 2814.5546875* proc.memory.percent‚17.178678512573242 cpu‚25.2309931485741( disk.nvme0c0n1.out‚15543.55908203125& disk.nvme0n1.out‚15543.55908203125 disk.dm-1.in‚0 disk.dm-0.in‚ 1964.80859375 disk.sda.out‚ 26.30078125 disk.sda3.out‚ 26.30078125$ memory_percent‚96.57735824584961Ôñ ‘ž‚(:† ½ÙêÔÆ…¾ö gpu.0.gpu‚24.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚102.302' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.4604 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0.ÇÝTŸ‚(:É ½ÙêÔÖì¶û# disk.dm-0.out‚15778.47314453125 disk.sda.in‚0 disk.sda3.out‚26.625$ memory_percent‚96.58195972442627( proc.memory.availableMB‚ 560.01171875 network.recv‚366467 disk.nvme0c0n1.in‚ 2101.6875 disk.drbd1260.in‚0 disk.sda2.out‚0 disk.sda.out‚26.625 cpu‚25.229571439239372) disk./.usagePercent‚78.87013117703171 disk.nvme0n1.in‚ 2101.6875 disk.sda3.in‚0" proc.memory.rssMB‚ 2814.5546875* proc.memory.percent‚17.178678512573242 proc.cpu.threads‚17 network.sent‚877435% disk./.usageGB‚1175.2219505310059( disk.nvme0c0n1.out‚15778.47314453125& disk.nvme0n1.out‚15778.47314453125 disk.dm-1.in‚0 disk.dm-1.out‚26.625 disk.drbd1260.out‚0.125 disk.sda2.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 2101.6875Ï ( ‚(:… ÌÙêÔ¶ž¾ø gpu.0.gpu‚18.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚103.195' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.639 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0-WáõP¡‚(:Å ÌÙêÔ¨øÿ& disk.nvme0n1.out‚15879.55810546875 disk.dm-1.out‚27.25 proc.cpu.threads‚17 disk.dm-0.in‚ 2101.6875 disk.sda.in‚0 disk.sda.out‚27.25 disk.nvme0c0n1.in‚ 2101.6875 disk.dm-1.in‚0 disk.drbd1260.in‚0$ memory_percent‚96.58050537109375" proc.memory.availableMB‚560.25% disk./.usageGB‚1174.7779006958008( disk.nvme0c0n1.out‚15879.55810546875 disk.sda1.in‚0 disk.drbd1260.out‚ 0.12890625" proc.memory.rssMB‚ 2814.5546875* proc.memory.percent‚17.178678512573242 cpu‚25.228610419410252 network.sent‚881816 network.recv‚371164) disk./.usagePercent‚78.84033062000847 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.nvme0n1.in‚ 2101.6875 disk.sda1.out‚0# disk.dm-0.out‚15879.55810546875 disk.sda3.out‚27.25#›Ö¦œ¢‚(:‘ ÛÙêÔÙ®Àú gpu.0.gpu‚16.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚137.658' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.531599999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÕÕ€b£‚(:× ÛÙêÔýéãû network.sent‚886739) disk./.usagePercent‚78.85196232232667 disk.sda2.in‚0 disk.sda.in‚0 disk.sda3.in‚0 disk.dm-1.out‚ 27.609375$ memory_percent‚96.57905101776123* proc.memory.percent‚17.178678512573242 network.recv‚374499% disk./.usageGB‚1174.9512214660645 disk.drbd1260.in‚0 disk.sda2.out‚0# disk.dm-0.out‚16336.16748046875 disk.sda.out‚ 27.609375 disk.sda3.out‚ 27.609375( disk.nvme0c0n1.out‚16336.16748046875 disk.drbd1260.out‚ 0.12890625 disk.nvme0n1.in‚ 2101.6875& disk.nvme0n1.out‚16336.16748046875 disk.sda1.out‚0 disk.dm-1.in‚0( proc.memory.availableMB‚ 560.48828125" proc.memory.rssMB‚ 2814.5546875 proc.cpu.threads‚17 disk.dm-0.in‚ 2101.6875 disk.nvme0c0n1.in‚ 2101.6875 disk.sda1.in‚0 cpu‚25.2263603800358189jiS0¤+ _wandbruntime‚939  _runtime‚939Õ0·DÄ¥¾ & lr-AdamW/pg1‚0.0004172586642748802 & lr-AdamW/pg2‚0.0004172586642748802 trainer/global_step‚549 % _timestamp‚1.7885217007860098e+09 _runtime‚ 939.005979608 _step‚40þ¹ïûȦ % _timestamp‚1.7885217007860098e+09 _runtime‚ 939.005979608 _step‚40 & lr-AdamW/pg1‚0.0004172586642748802 & lr-AdamW/pg2‚0.0004172586642748802 trainer/global_step‚549(–wâè0§+ _wandbruntime‚939  _runtime‚939‡Þ*©¨¹O TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10063741 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1079300776448 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 327892480 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚22704605832704 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9780833 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚22704605832704 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10063739 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 276706304 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3830312 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1290 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1076247862376 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚22704354928128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚22943306927616 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚21625054151680 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚703 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1099971391488 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚21731779400312 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 % _timestamp‚1.7885217007877035e+09 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _step‚41 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚22704354928128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚10063739 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚14730866 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1079533678080 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1079300776448 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚21625072154624 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚21625072154624 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9780130ø¼!#Ã'©½O UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚14706020 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚24042977672704 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚22943282987520 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1076476835912 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚24794607 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 300646400 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1099694685184 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚14706020 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚22704605832704 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9780833 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚22704605832704 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10063739 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 276706304 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3830312 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1290 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1076247862376 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚22704354928128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚22943306927616 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚21625054151680 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚703 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1099971391488 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚21731779400312 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 % _timestamp‚1.7885217007877035e+09 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _step‚41 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚22704354928128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚10063739 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚14730866 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1079533678080 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1079300776448 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚21625072154624 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚21625072154624 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9780130 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10063741 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1079300776448 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 327892480 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 IDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24839600 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 311118848 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚14757584 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9798302 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1078435325106 % _timestamp‚1.7885217017910333e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚14732712 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 327892480 _step‚42 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚24814726 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚22745604990976 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1081497717248 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚21771268246930 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3836970 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1101975996416 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10082014 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚22745855908864 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚21664358191616 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚21664340188672 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚20692815882464 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚21664358191616 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚24814726 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1309 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚20692832921824 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1081264802304 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚14757584 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚22984990897664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1078206351362 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 _runtime‚ 940.012272172 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1081264802304 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9797590 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5960622 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚24086966894080 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10082016 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚7129˜h) ¬ÇM UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1081264802304 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9797590 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5960622 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚24086966894080 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10082016 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚712 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24839600 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 311118848 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚14757584 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9798302 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1078435325106 % _timestamp‚1.7885217017910333e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚14732712 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 327892480 _step‚42 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚24814726 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚22745604990976 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1081497717248 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚21771268246930 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3836970 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1101975996416 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10082014 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytesY+ð…Â.all.allocated‚22745855908864 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚21664358191616 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚21664340188672 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚20692815882464 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚21664358191616 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚24814726 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1309 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚20692832921824 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1081264802304 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚14757584 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚22984990897664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1078206351362 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 _runtime‚ 940.012272172 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0*Û# Î0­+ _wandbruntime‚940  _runtime‚940¹‚R®Ì -loss/FBt3/detection_loss‚8.58793443441391 /loss/FRt2/assignment_loss‚6.161057853698731 2loss/SRqqt2/assignment_loss‚2.6551791144840755 1loss/FBt4/assignment_loss‚0.25077689116123775 /loss/FRt4/detection_loss‚3.3339760341951923 /loss/FRt3/detection_loss‚3.0249442345566218 epoch‚0 1loss/SRqqt2/detection_loss‚1.5647706933859546 0loss/FBt3/assignment_loss‚0.5254503140846888 _runtime‚ 940.014925464 0loss/SRqqt1/detection_loss‚10.87420161565145 0loss/SRqqt4/detection_loss‚4.468858629465103 .loss/FBt1/detection_loss‚4.358125124288642 %loss/total_loss‚89.71361935898675 0loss/FRt1/assignment_loss‚6.5112286938561335 /loss/FRt4/assignment_loss‚6.879547472923033 1loss/SRqqt1/assignment_loss‚6.571946938832601 0loss/FBt2/assignment_loss‚0.5382121077186879 2loss/SRqqt3/assignment_loss‚2.6280238209886755 2loss/SRqqt4/assignment_loss‚3.4327872666445645 0loss/FBt1/assignment_loss‚0.7008351620448672 /loss/FRt2/detection_loss‚2.0895753954138074 .loss/FRt1/detection_loss‚2.651273078388638 /loss/FBt4/detection_loss‚3.5150660276412964 _step‚43 /loss/FRt3/assignment_loss‚4.955310376485189 1loss/SRqqt3/detection_loss‚1.3692702220633943 .loss/FBt2/detection_loss‚2.065267856600261 trainer/global_step‚549 % _timestamp‚1.7885217017954905e+09^þ…~V¯Ð /loss/FRt3/assignment_loss‚4.955310376485189 1loss/SRqqt3/detection_loss‚1.3692702220633943 .loss/FBt2/detection_loss‚2.065267856600261 trainer/global_step‚549 % _timestamp‚1.7885217017954905e+09 -loss/FBt3/detection_loss‚8.58793443441391 /loss/FRt2/assignment_loss‚6.161057853698731 2loss/SRqqt2/assignment_loss‚2.6551791144840755 1loss/FBt4/assignment_loss‚0.25077689116123775 /loss/FRt4/detection_loss‚3.3339760341951923 /loss/FRt3/detection_loss‚3.0249442345566218 epoch‚0 1loss/SRqqt2/detection_loss‚1.5647706933859546 0loss/FBt3/assignment_loss‚0.5254503140846888 _runtime‚ 940.014925464 0loss/SRqqt1/detection_loss‚10.87420161565145 0loss/SRqqt4/detection_loss‚4.468858629465103 .loss/FBt1/detection_loss‚4.358125124288642 %loss/total_loss‚89.71361935898675 0loss/FRt1/assignment_loss‚6.5112286938561335 /loss/FRt4/assignment_loss‚6.879547472923033 1loss/SRqqt1/assignment_loss‚6.571946938832601 0loss/FBt2/assignment_loss‚0.5382121077186879 2loss/SRqqt3/assignment_loss‚2.6280238209886755 2loss/SRqqt4/assignment_loss‚3.4327872666445645 0loss/FBt1/assignment_loss‚0.7008351620448672 /loss/FRt2/detection_loss‚2.0895753954138074 .loss/FRt1/detection_loss‚2.651273078388638 /loss/FBt4/detection_loss‚3.5150660276412964 _step‚43+JâSPg°‚(:Ü êÙêÔÆ»×ø& disk.nvme0n1.out‚16679.73779296875 disk.drbd1260.in‚0 disk.sda.out‚ 28.20703125 disk.sda3.out‚ 28.20703125$ memory_percent‚96.57671451568604) disk./.usagePercent‚78.85040272442808 disk.dm-1.out‚ 28.20703125 disk.sda2.in‚0 disk.dm-0.in‚ 2101.6875 disk.sda3.in‚0( proc.memory.availableMB‚ 560.87109375 cpu‚25.222858288663705% disk./.usageGB‚1174.9279823303223 disk.sda1.in‚0 disk.dm-1.in‚0" proc.memory.rssMB‚ 2814.5546875* proc.memory.percent‚17.178678512573242 proc.cpu.threads‚17 network.sent‚891901 disk.nvme0c0n1.in‚ 2101.6875( disk.nvme0c0n1.out‚16679.73779296875 disk.sda1.out‚0 disk.drbd1260.out‚ 0.1328125 disk.sda2.out‚0# disk.dm-0.out‚16679.73779296875 disk.sda.in‚0 network.recv‚378528 disk.nvme0n1.in‚ 2101.6875®$õ‘±‚(:† êÙêÔµßÔú gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚116.868' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.3736 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0æåpÆ›²‚(: ùÙêÔöãé÷ gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚33.0 gpu.0.powerWatts‚139.08' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.816000000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©¾p³‚(:å ùÙêÔˆÂïù" proc.memory.rssMB‚ 2814.5546875* proc.memory.percent‚17.178678512573242 disk.sda2.out‚0 disk.dm-0.in‚ 2122.16796875 disk.sda.out‚ 28.5703125& disk.nvme0n1.out‚16848.19482421875 proc.cpu.threads‚17 network.recv‚384025) disk./.usagePercent‚78.88567211439776% disk./.usageGB‚1175.4535217285156 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.1328125# disk.dm-0.out‚16848.19482421875 disk.sda.in‚0 disk.sda1.in‚0 network.sent‚941379 disk.sda3.in‚0! disk.nvme0n1.in‚ 2122.16796875 disk.sda1.out‚0$ memory_percent‚96.58138751983643( proc.memory.availableMB‚ 560.10546875 cpu‚25.220231986639998 disk.dm-1.in‚0 disk.dm-1.out‚ 28.5703125 disk.sda2.in‚0 disk.sda3.out‚ 28.5703125# disk.nvme0c0n1.in‚ 2122.16796875( disk.nvme0c0n1.out‚16848.19482421875£<¶Ö‘´‚(:† ˆÚêÔ¾°Ê÷ gpu.0.gpu‚27.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚111.396' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.2792 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0*kUÎkµ‚(:à ˆÚêÔ“Ö”ú disk.dm-1.in‚0 disk.dm-1.out‚ 28.98828125 disk.drbd1260.out‚ 0.13671875 disk.sda.out‚ 28.98828125 network.sent‚947588% disk./.usageGB‚1175.6089820861816& disk.nvme0n1.out‚17273.01123046875 disk.sda3.out‚ 28.98828125" proc.memory.rssMB‚ 2814.5546875* proc.memory.percent‚17.178678512573242 proc.cpu.threads‚17 disk.sda1.in‚0 disk.drbd1260.in‚0 disk.sda2.out‚0# disk.dm-0.out‚17273.01123046875 disk.sda.in‚0$ memory_percent‚96.57986164093018 network.recv‚389011 disk.nvme0c0n1.in‚ 2236.34375 disk.sda2.in‚0 disk.dm-0.in‚ 2236.34375 disk.sda3.in‚0( proc.memory.availableMB‚ 560.35546875 cpu‚25.217972933171367) disk./.usagePercent‚78.89610518944066( disk.nvme0c0n1.out‚17273.01123046875 disk.nvme0n1.in‚ 2236.34375 disk.sda1.out‚0Äo8¨h¶‚(:Ý —ÚêÔ俸ø# disk.dm-0.out‚17273.26513671875 disk.sda.in‚0 disk.sda.out‚ 29.58203125 disk.sda3.in‚0" memory_percent‚96.579909324646 network.recv‚392672 disk.sda3.out‚ 29.58203125& disk.nvme0n1.out‚17273.26513671875 disk.dm-1.in‚0 disk.dm-1.out‚ 29.58203125 disk.drbd1260.in‚0 proc.cpu.threads‚17 network.sent‚952815 disk.sda2.in‚0 disk.nvme0c0n1.in‚ 2236.34375 disk.nvme0n1.in‚ 2236.34375 disk.sda1.in‚0( proc.memory.availableMB‚ 560.34765625" proc.memory.rssMB‚ 2814.5546875* proc.memory.percent‚17.178678512573242$ disk./.usageGB‚1175.609001159668 disk.dm-0.in‚ 2236.34375( disk.nvme0c0n1.out‚17273.26513671875 disk.sda1.out‚0 disk.drbd1260.out‚ 0.13671875 cpu‚25.216363598408535) disk./.usagePercent‚78.89610646947833 disk.sda2.out‚0h:üî·‚(:„ —ÚêÔÜðÄ÷ gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.031' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.6062 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0-E`2¸- _wandbruntime‚1002  _runtime‚1002u¬)’Ź¿ & lr-AdamW/pg2‚0.0004552603641177655 trainer/global_step‚599 % _timestamp‚1.7885217644763494e+09 _runtime‚1002.696084875 _step‚44 & lr-AdamW/pg1‚0.0004552603641177655_K dɺà & lr-AdamW/pg1‚0.0004552603641177655 & lr-AdamW/pg2‚0.0004552603641177655 trainer/global_step‚599 % _timestamp‚1.7885217644763494e+09 _runtime‚1002.696084875 _step‚44,Y2æ*2»- _wandbruntime‚1002  _runtime‚1002öºF†Â'¼¼O cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1199970643456 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚45 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚23589356001280 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚16068818 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚23589374004224 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10668826 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚24766872484352 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚23589356001280 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1177516483072 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚689 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚16043972 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1174414876420 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1309 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚26227476073472 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 280900608 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10669517 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6489918 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚691 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 trainer/global_step‚599 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚24767123388928 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10977489 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 % _timestamp‚1.7885217644780402e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚10977491 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1177516483072 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 304840704 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚23589374004224 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚16068818 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚25027481489920 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 _runtime‚1002.698434714 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1177749384704 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚22531820435600 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚24766872484352 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4178908 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 300049920 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚24767123388928 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚27021461 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚16043972 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚25027505430016 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚23705989299124 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚10977489 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚27046309 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 % _timestamp‚1.7885217644780402e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚10977491 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1177516483072 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 304840704 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚23589374004224 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚16068818 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚25027481489920 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 _runtime‚1002.698434714 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1177749384704 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚22531820435600 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚24766872484352 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4178908 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 300049920 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚24767123388928 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚27021461 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚16043972 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚25027505430016 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚23705989299124 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚10977489 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚27046309 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚45 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚23589356001280 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚16068818 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚23589374004224 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10668826 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚24766872484352 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚23589356001280 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1177516483072 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚689 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚16043972 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1174414876420 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1309 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚26227476073472 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 280900608 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10669517 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6489918 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚691 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 trainer/global_step‚599-%“ ˜2¾- _wandbruntime‚1004  _runtime‚1004›Ö'8¿ÆM SEDeviceStatsMonitor.on_train_batch_end/requesteNn€!”&d_bytes.small_pool.peak‚ 560281229 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚16095536 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚24808373465088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1179713423872 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚711 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚23628642038272 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10687586 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1179713423872 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚24808122547200 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚16070664 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10995766 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚27091302 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚27066428 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1176144391870 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1309 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 287178752 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _step‚46 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚23745478145534 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1179480508928 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚23628642038272 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1201969972736 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚26271159372800 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚23628660041216 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6500722 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10995766 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚16070664 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚24808373465088 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚713 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6500720 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 311118848 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚25069165459968 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 trainer/global_step‚599 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚27091302 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚22569333753664 % _timestamp‚1.7885217660369995e+09 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4186153 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 _runtime‚1004.260517174 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚26270848253952 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚25069189400064 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚23628660041216 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10686873 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4186864 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1290 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10995764 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚16095536 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 300049920 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273ʉ©€Ð&ÀÊM [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 287178752 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _step‚46 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚23745478145534 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1179480508928 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚23628642038272 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1201969972736 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚26271159372800 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚23628660041216 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6500722 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10995766 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚16070664 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚24808373465088 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚713 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6500720 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 311118848 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚25069165459968 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 trainer/global_step‚599 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚27091302 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚22569333753664 % _timestamp‚1.7885217660369995e+09 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4186153 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 _runtime‚1004.260517174 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚26270848253952 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚25069189400064 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚23628660041216 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10686873 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4186864 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1290 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10995764 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚16095536 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 300049920 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚16095536 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚24808373465088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1179713423872 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚711 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚23628642038272 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10687586 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1179713423872 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚24808122547200 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚16070664 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10995766 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚27091302 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚27066428 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1176144391870 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1309.1&]^2Á- _wandbruntime‚1004  _runtime‚1004dÌ-2QÂË 2loss/SRqqt1/assignment_loss‚3.5857562101804294 1loss/SRqqt3/assignment_loss‚2.725780126923009 2loss/SRqqt4/assignment_loss‚3.3614659547805785 /loss/FRt1/detection_loss‚2.9558454638435725 _step‚47 /loss/FRt1/assignment_loss‚6.007948671068464 2loss/SRqqt2/assignment_loss‚2.4625710906920495 .loss/SRqqt2/detection_loss‚1.4556435053662 1loss/SRqqt3/detection_loss‚1.5013567922146698 epoch‚0 /loss/FRt2/assignment_loss‚5.249855487377612 0loss/FBt2/assignment_loss‚0.6923162301827688 .loss/FRt2/detection_loss‚1.997483109498953 _runtime‚1004.263212564 /loss/FRt4/assignment_loss‚6.698386684060097 -loss/FBt1/detection_loss‚3.73689128593965 /loss/FBt2/detection_loss‚2.3499346009533233 /loss/FRt4/detection_loss‚3.0231220051646233 0loss/SRqqt4/detection_loss‚3.079025583465894 0loss/FBt1/assignment_loss‚0.6364074957947113 0loss/FBt3/assignment_loss‚0.7229874134063721 0loss/FBt4/assignment_loss‚0.3751074852259626 /loss/FBt4/detection_loss‚2.3305685448646547 /loss/FRt3/assignment_loss‚5.318188589010665 /loss/FBt3/detection_loss‚21.407064199447632 trainer/global_step‚599 /loss/FRt3/detection_loss‚2.1110586995509135 1loss/SRqqt1/detection_loss‚4.3631201065503635 %loss/total_loss‚88.14788533556319 % _timestamp‚1.7885217660430672e+09 ¡©ÙUÃÏ 0loss/FBt3/assignment_loss‚0.7229874134063721 0loss/FBt4/assignment_loss‚0.3751074852259626 /loss/FBt4/detection_loss‚2.3305685448646547 /loss/FRt3/assignment_loss‚5.318188589010665 /loss/FBt3/detection_loss‚21.407064199447632 trainer/global_step‚599 /loss/FRt3/detection_loss‚2.1110586995509135 1loss/SRqqt1/detection_loss‚4.3631201065503635 %loss/total_loss‚88.14788533556319 % _timestamp‚1.7885217660430672e+09 2loss/SRqqt1/assignment_loss‚3.5857562101804294 1loss/SRqqt3/assignment_loss‚2.725780126923009 2loss/SRqqt4/assignment_loss‚3.3614659547805785 /loss/FRt1/detection_loss‚2.9558454638435725 _step‚47 /loss/FRt1/assignment_loss‚6.007948671068464 2loss/SRqqt2/assignment_loss‚2.4625710906920495 .loss/SRqqt2/detection_loss‚1.4556435053662 1loss/SRqqt3/detection_loss‚1.5013567922146698 epoch‚0 /loss/FRt2/assignment_loss‚5.249855487377612 0loss/FBt2/assignment_loss‚0.6923162301827688 .loss/FRt2/detection_loss‚1.997483109498953 _runtime‚1004.263212564 /loss/FRt4/assignment_loss‚6.698386684060097 -loss/FBt1/detection_loss‚3.73689128593965 /loss/FBt2/detection_loss‚2.3499346009533233 /loss/FRt4/detection_loss‚3.0231220051646233 0loss/SRqqt4/detection_loss‚3.079025583465894 0loss/FBt1/assignment_loss‚0.6364074957947113/8¸Ä‚(:… ¦ÚêÔ…Ðûù gpu.0.gpu‚28.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚98.782' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.7564 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0q|ðÜcÅ‚(:Ø ¦ÚêÔ˜åÈý disk.sda3.out‚ 29.90234375' disk.nvme0c0n1.out‚17705.0927734375 disk.dm-1.in‚0 proc.cpu.threads‚17 disk.sda.in‚0 disk.sda3.in‚0 disk.nvme0c0n1.in‚ 2236.34375% disk.nvme0n1.out‚17705.0927734375 disk.dm-1.out‚ 29.90234375$ memory_percent‚96.57878875732422* proc.memory.percent‚17.178678512573242 cpu‚25.214927690039218 disk.dm-0.in‚ 2236.34375" disk.dm-0.out‚17705.0927734375 disk.nvme0n1.in‚ 2236.34375 disk.sda1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚0.140625 network.sent‚957939% disk./.usageGB‚1175.6089820861816 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.out‚0% proc.memory.availableMB‚ 560.53125" proc.memory.rssMB‚ 2814.5546875 network.recv‚396363) disk./.usagePercent‚78.89610518944066 disk.sda.out‚ 29.90234375è¯b‘Æ‚(:† µÚêÔôê‡ö gpu.0.gpu‚18.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚105.192' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.0384 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0nôiÇ‚(:Þ µÚêÔíÉ’ø$ memory_percent‚96.58186435699463* proc.memory.percent‚17.178678512573242" disk.dm-0.out‚17866.3857421875) disk./.usagePercent‚78.89642238277558% disk./.usageGB‚1175.6137084960938 disk.drbd1260.out‚0.140625 disk.dm-0.in‚ 2236.34375' disk.nvme0c0n1.out‚17866.3857421875 network.sent‚ 1.007485e+06 disk.sda2.in‚0 disk.sda3.in‚0% disk.nvme0n1.out‚17866.3857421875( proc.memory.availableMB‚ 560.02734375 cpu‚25.214048078351407 proc.cpu.threads‚17 network.recv‚401732 disk.sda1.out‚0 disk.dm-1.out‚ 30.1953125 disk.drbd1260.in‚0 disk.sda.in‚0 disk.nvme0c0n1.in‚ 2236.34375 disk.nvme0n1.in‚ 2236.34375" proc.memory.rssMB‚ 2814.5546875 disk.sda1.in‚0 disk.dm-1.in‚0 disk.sda2.out‚0 disk.sda.out‚ 30.1953125 disk.sda3.out‚ 30.1953125â¶VÈ‚(:„ ÄÚêÔ†šÀ÷ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚102.135' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.427 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0×ýƒoÉ‚(:ä ÄÚêÔûµ¨ú disk.sda3.in‚0 disk.sda3.out‚30.65625 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0' proc.memory.availableMB‚ 560.2109375" proc.memory.rssMB‚ 2814.5546875& disk.nvme0n1.out‚18137.80029296875 disk.sda1.in‚0 network.sent‚ 1.013691e+06 network.recv‚406687) disk./.usagePercent‚78.89484128024411% disk./.usageGB‚1175.5901489257812! disk.nvme0n1.in‚ 2383.24609375 disk.dm-1.out‚30.65625 disk.sda.in‚0 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.14453125 disk.dm-0.in‚ 2383.24609375# disk.dm-0.out‚18137.80029296875$ memory_percent‚96.58074378967285* proc.memory.percent‚17.178678512573242 cpu‚25.21282891278365 disk.sda.out‚30.65625# disk.nvme0c0n1.in‚ 2383.24609375( disk.nvme0c0n1.out‚18137.80029296875 disk.drbd1260.in‚0 proc.cpu.threads‚17èSªõÊ‚(:„ ÓÚêÔ¨ü¾÷ gpu.0.gpu‚23.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚103.24' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.648 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÌѦ®wË‚(:ì ÓÚêÔ†Üù disk.sda1.in‚0 disk.drbd1260.in‚0 disk.sda3.out‚ 31.15234375# memory_percent‚96.5822696685791' proc.memory.availableMB‚ 559.9609375 cpu‚25.211348854306266 network.sent‚ 1.018636e+06& disk.nvme0n1.out‚18478.86962890625 disk.drbd1260.out‚ 0.14453125 disk.sda2.in‚0 disk.dm-0.in‚ 2383.24609375 disk.sda3.in‚0" proc.memory.rssMB‚ 2814.5546875 network.recv‚410036) disk./.usagePercent‚78.90062577048027# disk.nvme0c0n1.in‚ 2383.24609375( disk.nvme0c0n1.out‚18478.86962890625! disk.nvme0n1.in‚ 2383.24609375 disk.sda1.out‚0 disk.dm-1.out‚ 31.15234375 proc.cpu.threads‚17$ disk./.usageGB‚1175.676342010498 disk.dm-1.in‚0 disk.sda2.out‚0# disk.dm-0.out‚18478.86962890625 disk.sda.in‚0 disk.sda.out‚ 31.15234375* proc.memory.percent‚17.178678512573242twbÑ‘Ì‚(:† âÚêÔû ó÷ gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚109.628' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.9256 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ñ X6hÍ‚(:Ý âÚêÔ×Øø network.recv‚413253% disk.nvme0c0n1.out‚18820.62890625 disk.sda1.out‚0 disk.dm-1.out‚ 31.578125 disk.sda2.out‚0 disk.dm-0.in‚ 2383.24609375# proc.memory.rssMB‚ 2814.55859375* proc.memory.percent‚17.178702354431152 disk.sda.out‚ 31.578125 disk.sda3.out‚ 31.578125! disk.nvme0n1.in‚ 2383.24609375 disk.drbd1260.in‚0 network.sent‚ 1.023653e+06% disk./.usageGB‚1175.7388229370117 disk.sda3.in‚0# disk.nvme0n1.out‚18820.62890625 disk.sda1.in‚0 disk.sda2.in‚0 disk.dm-0.out‚18820.62890625& proc.memory.availableMB‚ 559.953125( disk./.usagePercent‚78.9048189178836 disk.sda.in‚0# disk.nvme0c0n1.in‚ 2383.24609375 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.1484375$ memory_percent‚96.58231735229492 cpu‚25.210343307168856 proc.cpu.threads‚17ñ^N›Î‚(: ñÚêÔ³‹÷ gpu.0.gpu‚19.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚29.0 gpu.0.powerWatts‚97.673' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚19.534599999999998 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õRŒ{mÏ‚(:â ñÚêÔ”¿Òú disk.drbd1260.in‚0* proc.memory.percent‚17.178702354431152 disk.dm-0.out‚19027.12890625# disk.nvme0c0n1.in‚ 2383.24609375% disk.nvme0c0n1.out‚19027.12890625$ memory_percent‚96.58083915710449' proc.memory.availableMB‚ 560.1953125 cpu‚25.208988912659837 network.sent‚ 1.028757e+06 disk.dm-0.in‚ 2383.24609375 disk.sda.in‚0 disk.sda3.in‚0! disk.nvme0n1.in‚ 2383.24609375# disk.nvme0n1.out‚19027.12890625 disk.dm-1.in‚0# proc.memory.rssMB‚ 2814.55859375 network.recv‚417295% disk./.usageGB‚1175.7385559082031 disk.sda.out‚ 31.9140625 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.out‚ 0.1484375 proc.cpu.threads‚17) disk./.usagePercent‚78.90480099735619 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 31.9140625 disk.dm-1.out‚ 31.9140625á'~2Ð- _wandbruntime‚1088  _runtime‚1088'ì`aÅÑ¿ & lr-AdamW/pg1‚0.0004932620639606506 & lr-AdamW/pg2‚0.0004932620639606506 trainer/global_step‚649 % _timestamp‚1.7885218504977605e+09 _runtime‚1088.718444204 _step‚48 ¹èrÉÒà & lr-AdamW/pg1‚0.0004932620639606506 & lr-AdamW/pg2‚0.0004932620639606506 trainer/global_step‚649 % _timestamp‚1.7885218504977605e+09 _runtime‚1088.718444204 _step‚480•Pˆ42Ó- _wandbruntime‚1088  _runtime‚10881úêœFÔ¼O D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚689 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1299747194880 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1275965091328 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚17381924 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1275732189696 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚29298011 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚27111679992320 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚26829640945152 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚25553675853824 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 LFDeviceStatsMonitor.on_train_batch_start/oversize_ak‹E| llocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11891239 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚25553657850880 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4527597 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚691 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1300042775552 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚24408338306800 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 319520768 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚17381924 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11557615 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4528286 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚26829390040576 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚26829390040576 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1275965091328 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚17381924 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1275732189696 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚29298011 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚27111679992320 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚26829640945152 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚25553675853824 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11891239 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚25553657850880 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4527597 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚691 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 13000427755521V%âþ2Ö- _wandbruntime‚1090  _runtime‚1090,vnÌ&×ÆM L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7040820 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 300049920 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11575074 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4534933 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚25719934057242 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚28455112837120 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚25592961890816 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1290 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚26870891021312 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚1090.840595677 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚25592943887872 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚25720180070346 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11909516 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 327892480 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚26870891021312 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚29343004 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1277929130496 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1302029762048 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7040822 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 304827392 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1274082432378 % _timestamp‚1.7885218526176825e+09 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚27153387902464 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚29318130 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚26870640103424 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚17433488 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚29343004 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11909516 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚679 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 280887296 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚17433488 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚28455417664512 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1277696215552 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚29318130 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11909514 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4534254 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11909514 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1309 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚29343004 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11909516 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚679 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 280887296 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚17433488 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚28455417664512 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1277696215552 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚29318130 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11909514 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4534254 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11909514 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1309 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7040820 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 300049920 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11575074 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4534933 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚25719934057242 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚28455112837120 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚25592961890816 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1290 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚26870891021312 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚1090.840595677 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚25592943887872 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚25720180070346 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11909516 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 327892480 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 5602812292³‰oE2Ù- _wandbruntime‚1090  _runtime‚1090ø1G#CÚ½ 2loss/SRqqt2/assignment_loss‚2.7113288932068405 loss/FBt3/assignment_loss‚0 epoch‚0 /loss/FRt3/assignment_loss‚5.524382680654526 .loss/FRt4/detection_loss‚3.148252429785552 2loss/SRqqt3/assignment_loss‚2.8786853965960053 1loss/SRqqt4/assignment_loss‚3.106153286420382 0loss/FBt2/assignment_loss‚0.6039018899847061 1loss/SRqqt2/detection_loss‚1.0565228800013184 trainer/global_step‚649 1loss/FBt1/assignment_loss‚0.34556717039438434 /loss/FRt2/detection_loss‚1.8491342957866819 1loss/SRqqt3/detection_loss‚1.2144763619099792 /loss/FRt1/assignment_loss‚6.373547405004501 1loss/FBt4/assignment_loss‚0.35667122755017605 /loss/FRt3/detection_loss‚2.5856660529971123 1loss/SRqqt4/detection_loss‚2.9965334213696995 /loss/FBt2/detection_loss‚2.0180588918256444 0loss/FRt2/assignment_loss‚5.9106870199504655 /loss/FRt4/assignment_loss‚6.613539095278139 /loss/SRqqt1/detection_loss‚4.62653648853302 #loss/total_loss‚74.277813111219 _runtime‚1090.843209283 /loss/FRt1/detection_loss‚3.2953090593218803 .loss/FBt1/detection_loss‚2.846791396880972 /loss/FBt4/detection_loss‚2.6206139093914698 % _timestamp‚1.7885218526232443e+09 _step‚51 2loss/SRqqt1/assignment_loss‚3.0987027883529663 .loss/FBt3/detection_loss‚8.496751070022583.ÛäGÛÁ 0loss/FBt2/assignment_loss‚0.6039018899847061 1loss/SRqqt2/detection_loss‚1.0565228800013184 trainer/global_step‚649 1loss/FBt1/assignment_loss‚0.34556717039438434 /loss/FRt2/detection_loss‚1.8491342957866819 1loss/SRqqt3/detection_loss‚1.2144763619099792 /loss/FRt1/assignment_loss‚6.373547405004501 1loss/FBt4/assignment_loss‚0.35667122755017605 /loss/FRt3/detection_loss‚2.5856660529971123 1loss/SRqqt4/detection_loss‚2.9965334213696995 /loss/FBt2/detection_loss‚2.0180588918256444 0loss/FRt2/assignment_loss‚5.9106870199504655 /loss/FRt4/assignment_loss‚6.613539095278139 /loss/SRqqt1/detection_loss‚4.62653648853302 #loss/total_loss‚74.277813111219 _runtime‚1090.843209283 /loss/FRt1/detection_loss‚3.2953090593218803 .loss/FBt1/detection_loss‚2.846791396880972 /loss/FBt4/detection_loss‚2.6206139093914698 % _timestamp‚1.7885218526232443e+09 _step‚51 2loss/SRqqt1/assignment_loss‚3.0987027883529663 .loss/FBt3/detection_loss‚8.496751070022583 2loss/SRqqt2/assignment_loss‚2.7113288932068405 loss/FBt3/assignment_loss‚0 epoch‚0 /loss/FRt3/assignment_loss‚5.524382680654526 .loss/FRt4/detection_loss‚3.148252429785552 2loss/SRqqt3/assignment_loss‚2.8786853965960053 1loss/SRqqt4/assignment_loss‚3.1061532864203823 šàüÜ‚(:… €ÛêÔäÒôö gpu.0.gpu‚21.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚29.0 gpu.0.powerWatts‚92.612' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.5224 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¢ ÍNyÝ‚(:î €ÛêÔÔ¥¹ø disk.dm-0.in‚ 2505.64453125' proc.memory.availableMB‚ 559.8828125 proc.cpu.threads‚17 disk.sda.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 32.31640625! disk.nvme0n1.in‚ 2505.64453125 disk.sda2.in‚0# proc.memory.rssMB‚ 2814.55859375* proc.memory.percent‚17.178702354431152 cpu‚25.208534010765106% disk./.usageGB‚1175.8405456542969 disk.sda.out‚ 32.31640625( disk.nvme0c0n1.out‚19344.78076171875& disk.nvme0n1.out‚19344.78076171875 disk.sda1.out‚0 disk.dm-1.out‚ 32.31640625 disk.drbd1260.in‚0 disk.sda2.out‚0 network.recv‚422063) disk./.usagePercent‚78.91164561479165# disk.nvme0c0n1.in‚ 2505.64453125 disk.sda1.in‚0 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.15234375# disk.dm-0.out‚19344.78076171875# memory_percent‚96.5827465057373 network.sent‚ 1.035071e+06=— ûœÞ‚(:‘ ÛêÔÉÅŽø gpu.0.gpu‚22.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚29.0 gpu.0.powerWatts‚100.254' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚20.050800000000002 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0—K”¾xß‚(:í ÛêÔÆîú disk.sda1.in‚0 disk.sda1.out‚0) disk./.usagePercent‚78.97902884586397% disk./.usageGB‚1176.8446044921875 disk.sda.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 32.73828125 disk.dm-1.in‚0& disk.nvme0n1.out‚19571.20654296875 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.dm-0.in‚ 2505.64453125# disk.dm-0.out‚19571.20654296875# proc.memory.rssMB‚ 2814.55859375 cpu‚25.20704693220928 network.sent‚ 1.083787e+06 network.recv‚426575( disk.nvme0c0n1.out‚19571.20654296875 disk.dm-1.out‚ 32.73828125 disk.drbd1260.out‚ 0.15234375 disk.sda2.out‚0$ memory_percent‚96.58079147338867& proc.memory.availableMB‚ 560.203125 disk.sda.out‚ 32.73828125# disk.nvme0c0n1.in‚ 2505.64453125! disk.nvme0n1.in‚ 2505.64453125* proc.memory.percent‚17.178702354431152 proc.cpu.threads‚17°÷¬Cà‚(:… žÛêÔè—¥ö gpu.0.gpu‚23.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚29.0 gpu.0.powerWatts‚96.099' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.2198 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0j„Û^á‚(:Ó žÛêÔ¾ñÚ÷# disk.nvme0c0n1.in‚ 2505.64453125# disk.nvme0c0n1.out‚ 20036.828125! disk.nvme0n1.out‚ 20036.828125 disk.dm-1.in‚0 disk.dm-1.out‚33.15625 disk.drbd1260.in‚0 disk.drbd1260.out‚0.15625 disk.sda.in‚0 network.sent‚ 1.08892e+06 network.recv‚430033% disk./.usageGB‚1177.0905494689941 disk.sda2.out‚0 disk.dm-0.out‚ 20036.828125 disk.sda.out‚33.15625 disk.sda3.out‚33.15625* proc.memory.percent‚17.178702354431152 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 2505.64453125 disk.sda3.in‚0$ memory_percent‚96.58083915710449# proc.memory.rssMB‚ 2814.55859375) disk./.usagePercent‚78.99553441961903! disk.nvme0n1.in‚ 2505.64453125 disk.sda2.in‚0' proc.memory.availableMB‚ 560.1953125 cpu‚25.206224337418238 proc.cpu.threads‚17)GÁ œâ‚(:‘ ­ÛêÔ™²üö gpu.0.gpu‚22.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚29.0 gpu.0.powerWatts‚102.183' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚20.436600000000002 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0—ªMÐdã‚(:Ù ­ÛêÔÐɪø# proc.memory.rssMB‚ 2814.55859375* proc.memory.percent‚17.178702354431152 network.sent‚ 1.096664e+06 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda.out‚ 33.62890625! disk.nvme0n1.in‚ 2505.64453125 disk.nvme0n1.out‚ 20205.28125 disk.drbd1260.out‚0.15625 cpu‚25.205068737548057 network.recv‚434388) disk./.usagePercent‚78.98672520036584 disk.sda1.out‚0 proc.cpu.threads‚17$ disk./.usageGB‚1176.959285736084 disk.sda.in‚0 disk.sda3.out‚ 33.62890625" disk.nvme0c0n1.out‚ 20205.28125 disk.sda1.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 33.62890625$ memory_percent‚96.58236503601074 disk.dm-0.in‚ 2505.64453125 disk.dm-0.out‚ 20205.28125 disk.sda3.in‚0# disk.nvme0c0n1.in‚ 2505.64453125 disk.drbd1260.in‚0' proc.memory.availableMB‚ 559.9453125xi9œä‚(:‘ ¼ÛêÔšÓƒ÷ gpu.0.gpu‚29.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚29.0 gpu.0.powerWatts‚109.885' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.977000000000004 gpu.0.smClock‚1290 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ò±,må‚(:â ¼ÛêÔÒÈçø disk.dm-1.out‚ 33.98046875 disk.drbd1260.in‚0 disk.sda2.in‚0 proc.cpu.threads‚17$ disk./.usageGB‚1177.021785736084 disk.sda.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 33.98046875( proc.memory.availableMB‚ 559.94140625 proc.memory.rssMB‚ 2814.5625 cpu‚25.203949132339268 disk.drbd1260.out‚ 0.16015625& disk.nvme0c0n1.out‚20550.787109375$ disk.nvme0n1.out‚20550.787109375 disk.sda1.out‚0 disk.dm-0.in‚ 2637.2578125 disk.sda.out‚ 33.98046875 network.sent‚ 1.102984e+06 network.recv‚439074" disk.nvme0c0n1.in‚ 2637.2578125 disk.dm-1.in‚0 disk.sda2.out‚0! disk.dm-0.out‚20550.787109375$ memory_percent‚96.58238887786865* proc.memory.percent‚17.178726196289062) disk./.usagePercent‚78.99091962780682 disk.nvme0n1.in‚ 2637.2578125 disk.sda1.in‚0=ûæ‚(:… ËÛêÔ£­¿ø gpu.0.gpu‚3.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚115.012' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.0024 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0儃jç‚(:ß ËÛêÔñë°ù disk.sda1.in‚0 network.sent‚ 1.107948e+06 disk.nvme0n1.in‚ 2637.2578125 proc.memory.rssMB‚ 2814.5625 cpu‚25.202852163953953 proc.cpu.threads‚17 network.recv‚442341 disk.drbd1260.out‚ 0.16015625 disk.sda2.in‚0 disk.dm-0.in‚ 2637.2578125 disk.sda.in‚0 disk.sda.out‚ 34.4140625& disk.nvme0c0n1.out‚20608.998046875$ disk.nvme0n1.out‚20608.998046875 disk.sda2.out‚0 disk.sda3.out‚ 34.4140625" disk.nvme0c0n1.in‚ 2637.2578125 disk.sda1.out‚0$ memory_percent‚96.58238887786865( proc.memory.availableMB‚ 559.94140625* proc.memory.percent‚17.178726196289062( disk./.usagePercent‚78.9909029873171% disk./.usageGB‚1177.0215377807617 disk.dm-1.in‚0 disk.dm-1.out‚ 34.4140625 disk.drbd1260.in‚0! disk.dm-0.out‚20608.998046875 disk.sda3.in‚0Y¯n'œè‚(:‘ ÚÛêÔƒÚ÷ gpu.0.gpu‚29.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚29.0 gpu.0.powerWatts‚104.543' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚20.908600000000003 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Jgjšlé‚(:á ÚÛêÔŽñƒû network.sent‚ 1.111242e+06 disk.dm-1.out‚ 35.0234375 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.1640625 disk.dm-0.in‚ 2637.2578125 disk.sda3.in‚0 disk.sda3.out‚ 35.0234375% disk.nvme0n1.out‚20926.4697265625) disk./.usagePercent‚78.99771560381078 disk.sda2.in‚0 disk.sda.in‚0 disk.sda1.out‚0' proc.memory.availableMB‚ 560.1640625 proc.memory.rssMB‚ 2814.5625* proc.memory.percent‚17.178726196289062 cpu‚25.200320886268813% disk./.usageGB‚1177.1230506896973 disk.sda.out‚ 35.0234375 disk.nvme0n1.in‚ 2637.2578125 proc.cpu.threads‚17 network.recv‚446197 disk.dm-1.in‚0 disk.sda2.out‚0" disk.dm-0.out‚20926.4697265625" disk.nvme0c0n1.in‚ 2637.2578125' disk.nvme0c0n1.out‚20926.4697265625 disk.sda1.in‚0$ memory_percent‚96.58102989196777É,€2ê- _wandbruntime‚1186  _runtime‚1186Œ>õåÃë½ % lr-AdamW/pg1‚0.000531263763803536 % lr-AdamW/pg2‚0.000531263763803536 trainer/global_step‚699 % _timestamp‚1.7885219479226418e+09 _runtime‚1186.142832756 _step‚52YÁžÇìÁ % lr-AdamW/pg1‚0.000531263763803536 % lr-AdamW/pg2‚0.000531263763803536 trainer/global_step‚699 % _timestamp‚1.7885219479226418e+09 _runtime‚1186.142832756 _step‚5242ÙFš2í- _wandbruntime‚1186  _runtime‚1186´«Ä.À'îºO XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚28892158501376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1374180797952 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12447877 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1399949615104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12804991 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1290 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚27654901122540 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚699 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1370290957436 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚30595852049920 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚27517959700480 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚29195878494720 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚18719876 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7570120 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4878463 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 300049920 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7570118 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 _runtime‚1186.144501242 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12804989 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚706 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1370061983900 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚18744722 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚27517977703424 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚31549713 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚26284839138640 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚28891907596800 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚18719876 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 287192064 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚12804991 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚704 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚31524865 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1373947896320 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1373947896320 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚30595540917760 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚27517959700480 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 # _timestamp‚1.78852194792441e+09 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 311132160 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚26284856178000 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚27655147135436 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚28892158501376 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚28891907596800 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚12804989 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚29195902434816 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12448583 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 327892480 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1399662423040 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608žH¥ï¾O H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚704 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚31524865 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1373947896320 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1373947896320 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚30595540917760 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚27517959700480 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 # _timestamp‚1.78852194792441e+09 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 311132160 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚26284856178000 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚27655147135436 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚28892158501376 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚28891907596800 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚12804989 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚29195902434816 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12448583 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 327892480 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1399662423040 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚28892158501376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1374180797952 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12447877 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1399949615104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12804991 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1290 =6DeviceStatsMonitorÊ|Q.on_train_batch_start/max_split_size‚-1 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚27654901122540 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚699 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1370290957436 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚30595852049920 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚27517959700480 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚29195878494720 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚18719876 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7570120 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4878463 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 300049920 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7570118 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 _runtime‚1186.144501242 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12804989 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚706 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1370061983900 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚18744722 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚27517977703424 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚31549713 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚26284839138640 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚28891907596800 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚18719876 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 287192064 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚12804991 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 5642588165úÕn2ð- _wandbruntime‚1187  _runtime‚1187éj…PÌ&ñÆM [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 295567360 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚31594706 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18771440 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12465992 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4884366 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1372020472886 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚18771440 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1401948026880 _step‚54 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12823264 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12465286 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4885070 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚31569832 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7885219496992602e+09 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚27557263740416 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12823264 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7580920 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚704 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚12823266 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1376144837120 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1290 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 319507456 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚29237562464768 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚27557245737472 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚30639534431744 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 300049920 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚18746568 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚28933408577536 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚28933408577536 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚28933157659648 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1375911922176 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096Ê…(åÐ&òÊM J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12823264 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12465286 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4885070 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚31569832 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7885219496992602e+09 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚27557263740416 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12823264 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7580920 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚704 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚12823266 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1376144837120 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1290 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 319507456 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚29237562464768 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚27557245737472 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚30639534431744 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 300049920 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚18746568 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚28933408577536 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚28933408577536 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚28933157659648 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1375911922176 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 295567360 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚31594706 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18771440 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12465992 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4884366 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1372020472886 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚18771440 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1401948026880 _step‚54 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚5826iL$2ó- _wandbruntime‚1187  _runtime‚1187»i@ôº /loss/FRt1/assignment_loss‚5.941462473435835 2loss/SRqqt4/assignment_loss‚2.3575973987579344 /loss/FRt1/detection_loss‚2.6664788831364024 0loss/SRqqt1/detection_loss‚5.208650330702464 .loss/FBt2/detection_loss‚4.590105454290077 _step‚55 0loss/FRt4/assignment_loss‚6.6127940741452305 1loss/SRqqt3/assignment_loss‚2.890083195181454 1loss/FBt4/assignment_loss‚0.35012164940248375 /loss/FBt1/detection_loss‚3.2308321708724614 .loss/FBt3/detection_loss‚4.487244606018066 trainer/global_step‚699 /loss/FRt3/assignment_loss‚5.473940962650737 0loss/SRqqt3/detection_loss‚1.375441425337511 2loss/SRqqt1/assignment_loss‚2.8285699685414634 /loss/FBt4/detection_loss‚1.8002049646207265 %loss/total_loss‚73.06244618329197 epoch‚0 % _timestamp‚1.7885219497020895e+09 0loss/SRqqt2/assignment_loss‚2.60365228903921 1loss/SRqqt2/detection_loss‚1.1041856256488245 0loss/SRqqt4/detection_loss‚4.524534958600998 _runtime‚1187.921436126 /loss/FRt2/assignment_loss‚6.778472010294596 0loss/FBt2/assignment_loss‚0.5196018160889625 0loss/FBt1/assignment_loss‚0.4726232707210199 loss/FBt3/assignment_loss‚0 /loss/FRt2/detection_loss‚1.8585455763339995 .loss/FRt3/detection_loss‚2.151432869864292 .loss/FRt4/detection_loss‚3.235870209607211ÔÉ[oDõ¾ /loss/FRt2/assignment_loss‚6.778472010294596 0loss/FBt2/assignment_loss‚0.5196018160889625 0loss/FBt1/assignment_loss‚0.4726232707210199 loss/FBt3/assignment_loss‚0 /loss/FRt2/detection_loss‚1.8585455763339995 .loss/FRt3/detection_loss‚2.151432869864292 .loss/FRt4/detection_loss‚3.235870209607211 /loss/FRt1/assignment_loss‚5.941462473435835 2loss/SRqqt4/assignment_loss‚2.3575973987579344 /loss/FRt1/detection_loss‚2.6664788831364024 0loss/SRqqt1/detection_loss‚5.208650330702464 .loss/FBt2/detection_loss‚4.590105454290077 _step‚55 0loss/FRt4/assignment_loss‚6.6127940741452305 1loss/SRqqt3/assignment_loss‚2.890083195181454 1loss/FBt4/assignment_loss‚0.35012164940248375 /loss/FBt1/detection_loss‚3.2308321708724614 .loss/FBt3/detection_loss‚4.487244606018066 trainer/global_step‚699 /loss/FRt3/assignment_loss‚5.473940962650737 0loss/SRqqt3/detection_loss‚1.375441425337511 2loss/SRqqt1/assignment_loss‚2.8285699685414634 /loss/FBt4/detection_loss‚1.8002049646207265 %loss/total_loss‚73.06244618329197 epoch‚0 % _timestamp‚1.7885219497020895e+09 0loss/SRqqt2/assignment_loss‚2.60365228903921 1loss/SRqqt2/detection_loss‚1.1041856256488245 0loss/SRqqt4/detection_loss‚4.524534958600998 _runtime‚1187.9214361267 “³ ö‚(:… éÛêÔÂûÜ÷ gpu.0.gpu‚8.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚29.0 gpu.0.powerWatts‚101.848' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.3696 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"G6¿p÷‚(:å éÛêÔû§Ùø network.sent‚ 1.117376e+06% disk./.usageGB‚1177.1235084533691' disk.nvme0c0n1.out‚21367.8212890625 disk.sda2.in‚0 disk.sda3.out‚ 35.35546875$ memory_percent‚96.58138751983643( proc.memory.availableMB‚ 560.10546875" disk.nvme0c0n1.in‚ 2637.2578125 disk.nvme0n1.in‚ 2637.2578125 disk.sda1.out‚0 disk.dm-0.in‚ 2637.2578125" disk.dm-0.out‚21367.8212890625 disk.sda.in‚0 network.recv‚450740) disk./.usagePercent‚78.99774632471488 disk.dm-1.in‚0 disk.dm-1.out‚ 35.35546875 disk.drbd1260.in‚0 proc.memory.rssMB‚ 2814.5625* proc.memory.percent‚17.178726196289062% disk.nvme0n1.out‚21367.8212890625 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda.out‚ 35.35546875 disk.sda3.in‚0 cpu‚25.199615406615838 proc.cpu.threads‚17 disk.drbd1260.out‚ 0.1640625ñ§ø‚(:… øÛêÔ®åö gpu.0.gpu‚23.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚29.0 gpu.0.powerWatts‚103.195' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.639 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ë òaù‚(:Ö øÛêÔíå÷$ memory_percent‚96.58141136169434& disk.nvme0c0n1.out‚21594.462890625$ disk.nvme0n1.out‚21594.462890625 disk.sda1.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚35.8125 disk.drbd1260.out‚ 0.16796875' proc.memory.availableMB‚ 560.1015625 proc.memory.rssMB‚ 2814.5625) disk./.usagePercent‚79.01452045037337 disk.sda.in‚0 disk.sda3.in‚0! disk.dm-0.out‚21594.462890625* proc.memory.percent‚17.178726196289062 proc.cpu.threads‚17 network.sent‚ 1.167455e+06 disk.sda.out‚35.8125 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 cpu‚25.198472653821913 network.recv‚456595% disk./.usageGB‚1177.3734550476074 disk.sda3.out‚35.8125" disk.nvme0c0n1.in‚ 2781.4921875 disk.nvme0n1.in‚ 2781.4921875 disk.sda1.in‚0 disk.dm-0.in‚ 2781.4921875MF}ú‚(:„ ‡ÜêÔ”íÎ÷ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚28.0 gpu.0.powerWatts‚92.504' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.5008 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0·Äiû‚(:Þ ‡ÜêÔá–û$ memory_percent‚96.58143520355225 proc.memory.rssMB‚ 2814.5625* proc.memory.percent‚17.178726196289062 disk.dm-1.out‚ 36.1484375 network.recv‚459959" disk.nvme0c0n1.in‚ 2781.4921875$ disk.nvme0n1.out‚22059.794921875 disk.sda.out‚ 36.1484375 disk.sda3.in‚0 disk.sda3.out‚ 36.1484375( proc.memory.availableMB‚ 560.09765625& disk.nvme0c0n1.out‚22059.794921875 disk.sda.in‚0 cpu‚25.197572301733263 proc.cpu.threads‚17 disk.drbd1260.out‚ 0.16796875 network.sent‚ 1.1723e+06) disk./.usagePercent‚79.01609668876168% disk./.usageGB‚1177.3969421386719 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 2781.4921875! disk.dm-0.out‚22059.794921875 disk.nvme0n1.in‚ 2781.4921875 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.sda2.out‚0e€±B‘ü‚(:† –ÜêÔ˺Ÿ÷ gpu.0.gpu‚24.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚29.0 gpu.0.powerWatts‚104.823' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.9646 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0´÷”êzý‚(:ã –ÜêÔ¾Öå÷ disk.nvme0n1.in‚ 2781.4921875& disk.nvme0n1.out‚22228.52001953125 disk.dm-1.in‚0 ×+qZôdisk.drbd1260.out‚0.171875 network.sent‚ 1.177583e+06) disk./.usagePercent‚78.95697456481176 disk.sda3.in‚0 disk.sda3.out‚ 36.8828125 disk.sda2.in‚0# proc.memory.rssMB‚ 2814.56640625 network.recv‚463429% disk./.usageGB‚1176.5159797668457 disk.sda.out‚ 36.8828125 disk.sda2.out‚0 disk.dm-0.in‚ 2781.4921875# memory_percent‚96.5829849243164% proc.memory.availableMB‚ 559.84375 disk.sda1.out‚0 disk.dm-1.out‚ 36.8828125 disk.drbd1260.in‚0# disk.dm-0.out‚22228.52001953125* proc.memory.percent‚17.178750038146973 cpu‚25.19696497701879 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda.in‚0" disk.nvme0c0n1.in‚ 2781.4921875( disk.nvme0c0n1.out‚22228.52001953125¤Ïþ‚(:… ¥ÜêÔ•¥Øö gpu.0.gpu‚10.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚29.0 gpu.0.powerWatts‚99.091' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.8182 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 z«sÿ‚(:è ¥ÜêÔÜ€÷ú disk.sda1.in‚0# proc.memory.rssMB‚ 2814.56640625 network.sent‚ 1.183309e+06 network.recv‚467539% disk./.usageGB‚1176.5159225463867 disk.sda2.out‚0 disk.dm-0.in‚ 2781.4921875" disk.nvme0c0n1.in‚ 2781.4921875& disk.nvme0n1.out‚22573.82080078125$ memory_percent‚96.58312797546387 disk.drbd1260.in‚0 disk.sda3.out‚ 37.16796875 disk.sda1.out‚0 disk.dm-1.in‚0 cpu‚25.1963030206248 proc.cpu.threads‚17) disk./.usagePercent‚78.95697072469875 disk.drbd1260.out‚0.171875 disk.sda.out‚ 37.16796875 disk.dm-1.out‚ 37.16796875' proc.memory.availableMB‚ 559.8203125* proc.memory.percent‚17.178750038146973 disk.sda2.in‚0# disk.dm-0.out‚22573.82080078125 disk.sda.in‚0 disk.sda3.in‚0( disk.nvme0c0n1.out‚22573.82080078125 disk.nvme0n1.in‚ 2781.4921875Ž¤;‡œ€‚(:‘ ´ÜêÔÂåòö gpu.0.gpu‚21.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚137.525' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.505000000000003 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0öp€óp‚(:å ´ÜêÔë×ø# memory_percent‚96.5831995010376# proc.memory.rssMB‚ 2814.56640625 cpu‚25.19570170940466) disk./.usagePercent‚78.95539960646111 disk.sda.in‚0 disk.sda.out‚ 37.61328125 disk.sda1.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 37.61328125 proc.cpu.threads‚17 disk.sda2.out‚0 disk.sda3.out‚ 37.61328125 disk.nvme0n1.in‚ 2831.8203125( proc.memory.availableMB‚ 559.80859375* proc.memory.percent‚17.178750038146973 network.recv‚472515 disk.sda2.in‚0! disk.dm-0.out‚22752.416015625" disk.nvme0c0n1.in‚ 2831.8203125& disk.nvme0c0n1.out‚22752.416015625$ disk.nvme0n1.out‚22752.416015625 disk.sda1.in‚0 disk.drbd1260.in‚0 network.sent‚ 1.191433e+06% disk./.usageGB‚1176.4925117492676 disk.dm-0.in‚ 2831.8203125 disk.sda3.in‚0 disk.drbd1260.out‚ 0.17578125®|½2‚- _wandbruntime‚1281  _runtime‚1281¸¶Åƒ¿ trainer/global_step‚749 % _timestamp‚1.7885220429507856e+09 _runtime‚1281.171252881 _step‚56 & lr-AdamW/pg1‚0.0005692654636464212 & lr-AdamW/pg2‚0.00056926546364642126êøÉ„Ã & lr-AdamW/pg1‚0.0005692654636464212 & lr-AdamW/pg2‚0.0005692654636464212 trainer/global_step‚749 % _timestamp‚1.7885220429507856e+09 _runtime‚1281.171252881 _step‚568U½l§2…- _wandbruntime‚1281  _runtime‚1281ê~?ÀÂ'†¼O UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚28161357009840 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13718741 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚29482279553024 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚20082674 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚29629603047144 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1468000024408 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚29482279553024 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚702 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5226161 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚31280100937216 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 _step‚57 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚29482261550080 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚20057828 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚30954425153024 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5226861 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚20057828 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1472396504576 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8110220 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚32780020423168 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 % _timestamp‚1.7885220429548545e+09 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1472163602944 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1499919485952 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚29482261550080 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚30954676057600 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚30954676057600 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚30954425153024 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚28161374049200 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1472396504576 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13337081 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1309 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 291386368 trainer/global_step‚749(.-Æ'‡ÀO UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚33801415 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13718739 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 315326464 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1499628099584 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚31280076997120 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1468228997944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚33776567 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13718739 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚29629357034248 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1309 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 291386368 trainer/global_step‚749 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚28161357009840 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13718741 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚29482279553024 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚20082674 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚29629603047144 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1468000024408 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚29482279553024 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚702 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5226161 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚31280100937216 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 _step‚57 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚29482261550080 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚20057828 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚30954425153024 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5226861 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚20057828 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1472396504576 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8110220 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚32780020423168 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 % _timestamp‚1.7885220429548545e+09 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1472163602944 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1499919485952 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚29482261550080 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚30954676057600 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚30954676057600 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚30954425153024 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚28161374049200 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1472396504576 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13337081 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 I6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚29521547587072 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1474360543744 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚20109392 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚28198887367264 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1501911303680 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚31321784907264 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚31321760967168 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13737016 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8121022 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚29521565590016 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚29668845880658 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5232739 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1469958513394 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13737016 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚33846408 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚20084520 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8121020 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5233435 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1470187487138 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 % _timestamp‚1.7885220452013233e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚20084520 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚33821534 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚33821534 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13737014 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚30995926133760 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚29521565590016 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13353759 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1309 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚749 _runtime‚1283.423827712 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1474127628800 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13354457 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚698 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚32823696210944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302147072Ž&}Ð&ŠÊM PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚28198887367264 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1501911303680 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚31321784907264 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚31321760967168 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13737016 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8121022 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚29521565590016 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚29668845880658 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5232739 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1469958513394 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13737016 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚33846408 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚20084520 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8121020 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5233435 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1470187487138 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 % _timestamp‚1.7885220452013233e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚20084520 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚33821534 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚33821534 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13737014 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚30995926133760 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚29521565590016 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13353759 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1309 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚749 _runtime‚1283.423827712 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1474127628800 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13354457 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚698 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚32823696210944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302147072 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 315313152 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚29521547587072 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1474360543744 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚20109392 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚34963881061888 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 302147072 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚34964190096896 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚33016942709248 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1570612211200 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚31603812945956 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1570379309568 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1290 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1565938064916 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14632491 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1566167038452 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 _runtime‚1379.330303947 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5573511 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚799 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14632489 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚21420626 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚702 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚33364275499520 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚30037891920400 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14632489 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1570379309568 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚34963881061888 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 302147072 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚34964190096896 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚33016942709248 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1570612211200 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚31603812945956 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1570379309568 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1290 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1565938064916 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14632491 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1566167038452 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 _runtime‚1379.330303947=Ñ3Üœ ‚(:‘ ÝêÔIJü gpu.0.gpu‚30.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚134.672' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.934399999999997 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Hpƒkj¡‚(:ß ÝêÔÁ¶ disk.sda3.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 network.recv‚501652 disk.sda.out‚ 40.44921875 disk.sda3.out‚ 40.44921875 disk.nvme0n1.in‚ 3124.8125 disk.drbd1260.in‚0# proc.memory.rssMB‚ 2814.56640625 cpu‚25.190617618057946# disk.dm-0.out‚24967.93017578125 disk.nvme0c0n1.in‚ 3124.8125( disk.nvme0c0n1.out‚24967.93017578125& disk.nvme0n1.out‚24967.93017578125 disk.sda1.in‚0 disk.dm-1.out‚ 40.44921875 disk.sda2.out‚0$ memory_percent‚96.58792018890381) disk./.usagePercent‚78.95698429309806 disk.dm-0.in‚ 3124.8125 disk.dm-1.in‚0 disk.drbd1260.out‚0.1875( proc.memory.availableMB‚ 559.03515625* proc.memory.percent‚17.178750038146973 proc.cpu.threads‚17 network.sent‚ 1.28468e+06% disk./.usageGB‚1176.5161247253418 disk.sda.in‚02Ç‚S2¢- _wandbruntime‚1380  _runtime‚1380Ïê/Ì&£ÆM M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚36098110 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚680 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 304827392 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚799 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14650764 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚31643301792366 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚682 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1290 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚35007588559360 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 _runtime‚1380.285685075 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14241795 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5581355 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5580675 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚35007893386752 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚30075422277824 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚36098110 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1568125527646 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1567896553902 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1572343335424 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14650766 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚36073236 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14650766 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 _step‚62 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚36073236 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚33405983409664 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1572343335424 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚30075405238464 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚31485867439616 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8661120 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚31643547805470 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚33058192772096 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚21447344 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1309 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1601909977088 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚33058443689984 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚33405959469568 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚31485849436672 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1572576250368 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 280887296 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1601629089792 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚21422472 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909<3ØwÐ&¤ÊM VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 % _timestamp‚1.7885221420636718e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚33058443689984 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚21447344 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚31485849436672 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8661122 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302147072 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14241795 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5581355 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5580675 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚35007893386752 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚30075422277824 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚36098110 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1568125527646 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1567896553902 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1572343335424 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14650766 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚36073236 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14650766 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 _step‚62 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚36073236 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚33405983409664 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1572343335424 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚30075405238464 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚31485867439616 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8661120 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚31643547805470 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚33058192772096 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚21447344 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1309 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1601909977088 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚33058443689984 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚33405959469568 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚31485849436672 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1572576250368 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 280887296 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1601629089792 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚21422472 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚36098110 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚680 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 304827392 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚799 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14650764 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚31643301792366 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚682 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1290 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚35007588559360 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 _runtime‚1380.285685075 OÉèI2¥- _wandbruntime‚1380  _runtime‚1380}ˆG¾D¦¾ % _timestamp‚1.7885221420688255e+09 1loss/SRqqt4/assignment_loss‚2.665122495757209 /loss/FRt3/detection_loss‚2.2084728826176034 /loss/FBt2/detection_loss‚2.4311872935895784 %loss/total_loss‚63.78108928014173 epoch‚0 2loss/SRqqt3/assignment_loss‚2.4758548435655614 0loss/FBt1/assignment_loss‚0.7349797046554158 /loss/FRt1/detection_loss‚2.8939074667600484 1loss/SRqqt2/detection_loss‚2.3749263709783555 0loss/SRqqt4/detection_loss‚4.068547381295098 loss/FBt3/assignment_loss‚0 /loss/FRt2/assignment_loss‚5.702446050811232 2loss/SRqqt1/assignment_loss‚3.0991196802684238 /loss/FRt4/detection_loss‚2.7454640929188048 0loss/FBt4/assignment_loss‚0.3910475630850052 _step‚63 2loss/SRqqt2/assignment_loss‚2.5678003311157225 0loss/FBt2/assignment_loss‚0.8499262239622314 0loss/SRqqt1/detection_loss‚3.626492949468749 _runtime‚1380.288615733 /loss/FRt3/assignment_loss‚4.685040374235673 /loss/FRt4/assignment_loss‚6.107454998152597 /loss/FBt1/detection_loss‚2.0974970768237937 /loss/FRt2/detection_loss‚2.2284236641829476 .loss/FRt1/assignment_loss‚5.91649893613962 1loss/SRqqt3/detection_loss‚1.1602471837936674 /loss/FBt3/detection_loss‚1.7724568843841553 /loss/FBt4/detection_loss‚0.9781748315802327 trainer/global_step‚799'OˆD§ 0loss/FBt4/ass(1ignment_loss‚0.3910475630850052 _step‚63 2loss/SRqqt2/assignment_loss‚2.5678003311157225 0loss/FBt2/assignment_loss‚0.8499262239622314 0loss/SRqqt1/detection_loss‚3.626492949468749 _runtime‚1380.288615733 /loss/FRt3/assignment_loss‚4.685040374235673 /loss/FRt4/assignment_loss‚6.107454998152597 /loss/FBt1/detection_loss‚2.0974970768237937 /loss/FRt2/detection_loss‚2.2284236641829476 .loss/FRt1/assignment_loss‚5.91649893613962 1loss/SRqqt3/detection_loss‚1.1602471837936674 /loss/FBt3/detection_loss‚1.7724568843841553 /loss/FBt4/detection_loss‚0.9781748315802327 trainer/global_step‚799 % _timestamp‚1.7885221420688255e+09 1loss/SRqqt4/assignment_loss‚2.665122495757209 /loss/FRt3/detection_loss‚2.2084728826176034 /loss/FBt2/detection_loss‚2.4311872935895784 %loss/total_loss‚63.78108928014173 epoch‚0 2loss/SRqqt3/assignment_loss‚2.4758548435655614 0loss/FBt1/assignment_loss‚0.7349797046554158 /loss/FRt1/detection_loss‚2.8939074667600484 1loss/SRqqt2/detection_loss‚2.3749263709783555 0loss/SRqqt4/detection_loss‚4.068547381295098 loss/FBt3/assignment_loss‚0 /loss/FRt2/assignment_loss‚5.702446050811232 2loss/SRqqt1/assignment_loss‚3.0991196802684238 /loss/FRt4/detection_loss‚2.7454640929188048?Fc…Ôœ¨‚(:‘ ¬ÝêÔÝ‹¤ö gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚114.614' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.922800000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»£x_©‚(:Ô ¬ÝêÔÇ’Ñý disk.drbd1260.out‚ 0.19140625 network.sent‚ 1.310334e+06 disk.sda2.out‚0$ memory_percent‚96.58825397491455* proc.memory.percent‚17.178750038146973) disk./.usagePercent‚78.95591494962756 disk.sda2.in‚0 disk.dm-0.out‚ 25395.8671875 disk.sda.out‚ 40.7421875 disk.sda3.in‚0 disk.nvme0c0n1.in‚ 3124.8125 disk.drbd1260.in‚0( proc.memory.availableMB‚ 558.98046875% disk./.usageGB‚1176.5001907348633 disk.sda3.out‚ 40.7421875 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 40.7421875# proc.memory.rssMB‚ 2814.56640625 cpu‚25.18998020982077 proc.cpu.threads‚17 network.recv‚507631 disk.dm-0.in‚ 3124.8125 disk.sda.in‚0$ disk.nvme0c0n1.out‚ 25395.8671875 disk.nvme0n1.in‚ 3124.8125" disk.nvme0n1.out‚ 25395.8671875 disk.dm-1.in‚0hà›ª‚(: »ÝêÔ‹¸¯÷ gpu.0.gpu‚19.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚29.0 gpu.0.powerWatts‚92.527' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚18.505399999999998 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¤±¹fe«‚(:Ú »ÝêÔñ£Ÿû$ memory_percent‚96.58379554748535! disk.nvme0n1.in‚ 3306.40234375 disk.dm-1.in‚0 disk.sda2.in‚0 disk.sda.out‚ 41.078125 disk.sda3.in‚0# disk.nvme0c0n1.in‚ 3306.40234375' proc.memory.availableMB‚ 559.7109375# proc.memory.rssMB‚ 2814.56640625 network.sent‚ 1.347566e+06 network.recv‚511231 disk.sda1.out‚0 disk.sda2.out‚0 disk.dm-0.out‚ 25692.921875# disk.nvme0c0n1.out‚ 25692.921875 proc.cpu.threads‚17% disk./.usageGB‚1176.5236778259277! disk.nvme0n1.out‚ 25692.921875 disk.sda3.out‚ 41.078125* proc.memory.percent‚17.178750038146973 cpu‚25.189656988599143) disk./.usagePercent‚78.95749118801587 disk.sda1.in‚0 disk.dm-1.out‚ 41.078125 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.19140625 disk.dm-0.in‚ 3306.40234375 disk.sda.in‚0,êg¬‚(:… ÊÝêÔ£×åö gpu.0.gpu‚9.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚29.0 gpu.0.powerWatts‚101.077' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.2154 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0óÓ·zw­‚(:ì ÊÝêÔéÙÄø network.recv‚515248% disk./.usageGB‚1176.5236587524414 disk.sda3.in‚0& disk.nvme0n1.out‚25871.63818359375 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0# memory_percent‚96.5853214263916 disk.sda2.out‚0# disk.dm-0.out‚25871.63818359375 disk.sda.out‚ 41.59765625# disk.nvme0c0n1.in‚ 3306.40234375( disk.nvme0c0n1.out‚25871.63818359375! disk.nvme0n1.in‚ 3306.40234375 disk.dm-1.out‚ 41.59765625 disk.drbd1260.in‚0 network.sent‚ 1.352859e+06( disk./.usagePercent‚78.9574899079782 disk.sda2.in‚0 disk.sda3.out‚ 41.59765625 disk.drbd1260.out‚ 0.1953125# proc.memory.rssMB‚ 2814.56640625* proc.memory.percent‚17.178750038146973 cpu‚25.189179564343622 disk.dm-0.in‚ 3306.40234375 disk.sda.in‚0' proc.memory.availableMB‚ 559.4609375 proc.cpu.threads‚17‰†R®‚(:„ ÙÝêÔ¼†½ø gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚29.0 gpu.0.powerWatts‚99.611' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.9222 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0@Ëîu¯‚(:ê ÙÝêÔȵÁù* proc.memory.percent‚17.178750038146973 cpu‚25.188416320309916 network.sent‚ 1.358636e+06 disk.drbd1260.in‚0# proc.memory.rssMB‚ 2814.56640625 proc.cpu.threads‚17 network.recv‚519387% disk./.usageGB‚1176.5001907348633 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda.out‚ 42.0078125 disk.sda3.out‚ 42.0078125( disk.nvme0c0n1.out‚26164.74365234375& disk.nvme0n1.out‚26164.74365234375) disk./.usagePercent‚78.95591494962756 disk.dm-0.in‚ 3306.40234375# disk.dm-0.out‚26164.74365234375 disk.sda.in‚0! disk.nvme0n1.in‚ 3306.40234375 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0 disk.sda3.in‚0# disk.nvme0c0n1.in‚ 3306.40234375 disk.dm-1.out‚ 42.0078125 disk.drbd1260.out‚ 0.1953125$ memory_percent‚96.58384323120117& proc.memory.availableMB‚ 559.703125u6îh°‚(:Ý èÝêԾČø( proc.memory.availableMB‚ 559.69921875# proc.memory.rssMB‚ 2814.56640625 network.sent‚ 1.364927e+06 network.recv‚524170) disk./.usagePercent‚78.95670294081793 disk.sda2.in‚0! disk.nvme0n1.in‚ 3306.40234375 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚96.58386707305908% disk.nvme0c0n1.out‚26481.87109375* proc.memory.percent‚17.178750038146973 cpu‚25.18775060555523 proc.cpu.threads‚17% disk./.usageGB‚1176.5119323730469 disk.drbd1260.out‚ 0.19921875 disk.sda2.out‚0 disk.dm-0.in‚ 3306.40234375 disk.sda.in‚0 disk.sda.out‚42.28125 disk.sda3.out‚42.28125# disk.nvme0n1.out‚26481.87109375 disk.drbd1260.in‚0 disk.dm-0.out‚26481.87109375 disk.sda3.in‚0# disk.nvme0c0n1.in‚ 3306.40234375 disk.dm-1.in‚0 disk.dm-1.out‚42.28125¹ÓZò±‚(:… èÝêÔ¡ó‡ù gpu.0.gpu‚21.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚29.0 gpu.0.powerWatts‚92.504' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.5008 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0äG‡g2²- _wandbruntime‚1469  _runtime‚1469ÖXjOų¿ & lr-AdamW/pg1‚0.0006452688633321917 & lr-AdamW/pg2‚0.0006452688633321917 trainer/global_step‚849 % _timestamp‚1.7885222315081193e+09 _runtime‚1469.728000084 _step‚64QŒ“[É´Ã % _timestamp‚1.7885222315081193e+09 _runtime‚1469.728000084 _step‚64 & lr-AdamW/pg1‚0.0006452688633321917 & lr-AdamW/pg2‚0.0006452688633321917 trainer/global_step‚849@†í}2µ- _wandbruntime‚1469  _runtime‚1469×Ð0Á'¶»O MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚35079460265472 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1668595016192 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 306937856 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚33578514870560 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 trainer/global_step‚849 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15111073 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 329989632 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚33578268857664 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚33410883252224 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚35448474001920 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15546239 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚702 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5920655 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15111777 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1668827917824 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 $ _timestamp‚1.788522231509598e+09 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚37148355637760 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9190418 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚31914392752240 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚33410865249280 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9190420 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1668827917824 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚22733732 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚704 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1699857695744 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚15546239 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1663876105424 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚15546241 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚22758578 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚35079460265472 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚35448497942016 _step‚65 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚22733732 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚33410865249280 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 _runtime‚1469.729275441 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚35079711170048 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944]³(Å'·¿O JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚704 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1699857695744 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚15546239 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1663876105424 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚15546241 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚22758578 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚35079460265472 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚35448497942016 _step‚65 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚22733732 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚33410865249280 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 _runtime‚1469.729275441 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚35079711170048 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚35079460265472 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1668595016192 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 306937856 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚33578514870560 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 trainer/global_step‚849 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15111073 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 329989632 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚33578268857664 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚33410883252224 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚35448474001920 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15546239 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚702 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5920655 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15111777 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1668827917824 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 $ _timestamp‚1.788522231509598e+09 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚37148355637760 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9190418 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚31914392752240 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚33410865249280 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9190420 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1668827917824 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚22733732A¯‘“Å‘¸‚(:† ÷ÝêÔù®¸ö gpu.0.gpu‚15.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚138.346' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.6692 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0X1nn¹‚(:ã ÷ÝêÔëãÂø* proc.memory.percent‚17.178750038146973 disk.sda1.out‚0 disk.dm-1.in‚0 network.recv‚527743% disk.nvme0c0n1.out‚26762.86328125 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.sda2.out‚0( proc.memory.availableMB‚ 559.63671875 disk.nvme0n1.in‚ 3441.6953125 disk.drbd1260.out‚ 0.19921875# proc.memory.rssMB‚ 2814.56640625 cpu‚25.18670676637372 proc.cpu.threads‚17 network.sent‚ 1.369818e+06) disk./.usagePercent‚78.94290387871486% disk./.usageGB‚1176.3063163757324 disk.sda3.out‚ 42.76953125" disk.nvme0c0n1.in‚ 3441.6953125 disk.sda1.in‚0 disk.dm-0.out‚26762.86328125 disk.sda.in‚0 disk.sda3.in‚0# disk.nvme0n1.out‚26762.86328125 disk.dm-1.out‚ 42.76953125 disk.dm-0.in‚ 3441.6953125 disk.sda.out‚ 42.76953125$ memory_percent‚96.58424854278564uÍÁÁ2º- _wandbruntime‚1470  _runtime‚1470Å,'• »ÆM VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚33450151286272 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚35120710328320 K>DeviceStatsMonitor.on|hX7_train_batch_end/allocation.all.allocated‚38349812 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚15564514 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15129355 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚37192035900416 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 300633088 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚31951940149024 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚38324938 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15564514 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚15564516 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15128672 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚38349812 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚33450169289216 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1670559042048 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302147072 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚22760424 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1670559042048 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚22785296 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚35490181912064 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚31951923109664 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚35120961246208 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚25âŒÐ&¼ÊM ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚31951940149024 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚38324938 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15564514 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚15564516 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15128672 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚38349812 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚33450169289216 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1670559042048 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302147072 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚22760424 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1670559042048 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚22785296 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚35490181912064 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚31951923109664 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚35120961246208 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚33450151286272 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚35120710328320 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚38349812 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚15564514 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15129355 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚37192035900416 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 300633088 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309BŽ «Û2½- _wandbruntime‚1470  _runtime‚1470°jmÜQ¾Ë /loss/FRt2/assignment_loss‚6.187212728318714 0loss/SRqqt2/detection_loss‚2.572616645693779 epoch‚0 /loss/FRt3/assignment_loss‚3.958402473276312 _runtime‚1470.758071361 2loss/SRqqt1/assignment_loss‚2.9717812299728394 2loss/SRqqt3/assignment_loss‚2.5899452738059585 0loss/FBt2/assignment_loss‚0.6577606309683209 1loss/FBt4/assignment_loss‚0.29261199013175115 /loss/FRt3/detection_loss‚2.1080860929055647 _step‚67 .loss/FBt3/assignment_loss‚0.80363529920578 1loss/SRqqt2/assignment_loss‚2.946583500504494 %loss/total_loss‚66.02947054665808 0loss/FBt1/assignment_loss‚0.7103129718288983 .loss/FRt4/detection_loss‚2.798580913774429 0loss/SRqqt1/detection_loss‚2.761665924390157 /loss/FBt4/detection_loss‚0.7871708962839188 % _timestamp‚1.7885222325384119e+09 /loss/FRt4/assignment_loss‚4.322543298044512 2loss/SRqqt4/assignment_loss‚2.9323681924078198 .loss/FRt2/detection_loss‚2.765733676297324 0loss/SRqqt4/detection_loss‚3.801818973488278 /loss/FBt2/detection_loss‚1.6783699801206067 .loss/FBt3/detection_loss‚5.681917428970337 trainer/global_step‚849 /loss/FRt1/assignment_loss‚6.716689971777109 /loss/FRt1/detection_loss‚2.6024670600891113 1loss/SRqqt3/detection_loss‚0.9137750485435475 /loss/FBt1/detection_loss‚2.4674203458585238 ’U¿Ï .loss/FRt4/detection_loss‚2.798580913774429 0loss/SRqqt1/detection_loss‚2.761665924390157 /loss/FBt4/detection_loss‚0.7871708962839188 % _timestamp‚1.7885222325384119e+09 /loss/FRt4/assignment_loss‚4.322543298044512 2loss/SRqqt4/assignment_loss‚2.9323681924078198 .loss/FRt2/detection_loss‚2.765733676297324 0loss/SRqqt4/detection_loss‚3.801818973488278 /loss/FBt2/detection_loss‚1.6783699801206067 .loss/FBt3/detection_loss‚5.681917428970337 trainer/global_step‚849 /loss/FRt1/assignment_loss‚6.716689971777109 /loss/FRt1/detection_loss‚2.6024670600891113 1loss/SRqqt3/detection_loss‚0.9137750485435475 /loss/FBt1/detection_loss‚2.4674203458585238 /loss/FRt2/assignment_loss‚6.187212728318714 0loss/SRqqt2/detection_loss‚2.572616645693779 epoch‚0 /loss/FRt3/assignment_loss‚3.958402473276312 _runtime‚1470.758071361 2loss/SRqqt1/assignment_loss‚2.9717812299728394 2loss/SRqqt3/assignment_loss‚2.5899452738059585 0loss/FBt2/assignment_loss‚0.6577606309683209 1loss/FBt4/assignment_loss‚0.29261199013175115 /loss/FRt3/detection_loss‚2.1080860929055647 _step‚67 .loss/FBt3/assignment_loss‚0.80363529920578 1loss/SRqqt2/assignment_loss‚2.946583500504494 %loss/total_loss‚66.02947054665808 0loss/FBt1/assignment_loss‚0.7103129718288983C6 E£‘À‚(:† †ÞêÔ–Œ®ø gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚131.433' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.2866 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Õ;ÚƒoÁ‚(:ä †ÞêÔ¹†Áù network.sent‚ 1.407512e+06) disk./.usagePercent‚78.90082238426656% disk.nvme0n1.out‚27295.4599609375 disk.dm-0.in‚ 3441.6953125" disk.dm-0.out‚27295.4599609375# proc.memory.rssMB‚ 2814.56640625* proc.memory.percent‚17.178750038146973 network.recv‚532336$ disk./.usageGB‚1175.679271697998 disk.sda1.out‚0 disk.dm-1.out‚ 43.1171875 disk.drbd1260.out‚ 0.20703125 disk.sda2.out‚0 disk.sda.out‚ 43.1171875 disk.sda3.in‚0 disk.nvme0n1.in‚ 3441.6953125 disk.sda1.in‚0 disk.dm-1.in‚0 disk.sda.in‚0" disk.nvme0c0n1.in‚ 3441.6953125' disk.nvme0c0n1.out‚27295.4599609375$ memory_percent‚96.58589363098145' proc.memory.availableMB‚ 559.3671875 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.sda3.out‚ 43.1171875 cpu‚25.18582659436535 proc.cpu.threads‚17+SF‘‚(:† •ÞêÔÁÓõ gpu.0.gpu‚29.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚109.921' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.9842 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ä4€/sÂ(:è •ÞêÔß…èü disk.sda.out‚ 43.65234375 disk.sda3.in‚0 disk.sda3.out‚ 43.65234375$ memory_percent‚96.58598899841309$ disk./.usageGB‚1175.679271697998' disk.nvme0c0n1.out‚27410.3896484375 disk.sda2.out‚0 disk.sda.in‚0' proc.memory.availableMB‚ 559.3515625* proc.memory.percent‚17.178750038146973 cpu‚25.184561918246782 network.sent‚ 1.448268e+06% disk.nvme0n1.out‚27410.3896484375 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.sda2.in‚0# proc.memory.rssMB‚ 2814.56640625) disk./.usagePercent‚78.90082238426656" disk.nvme0c0n1.in‚ 3441.6953125 disk.nvme0n1.in‚ 3441.6953125 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.20703125 disk.dm-0.in‚ 3441.6953125" disk.dm-0.out‚27410.3896484375 proc.cpu.threads‚17 network.recv‚537461 disk.dm-1.out‚ 43.65234375_!uAdÄ‚(:Ù ¤ÞêÔý¿ù" disk.nvme0c0n1.in‚ 3441.6953125 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 3441.6953125 disk.sda3.in‚0' proc.memory.availableMB‚ 559.6015625* proc.memory.percent‚17.178750038146973 network.sent‚ 1.45259e+06% disk./.usageGB‚1175.6792755126953 disk.dm-1.out‚44.1875 disk.sda2.in‚0 disk.sda.in‚0 disk.sda3.out‚44.1875 proc.cpu.threads‚17 disk.drbd1260.out‚ 0.2109375 network.recv‚542423( disk./.usagePercent‚78.9008226402741 disk.nvme0n1.in‚ 3441.6953125% disk.nvme0n1.out‚27559.3505859375 disk.dm-1.in‚0" disk.dm-0.out‚27559.3505859375 disk.sda.out‚44.1875' disk.nvme0c0n1.out‚27559.3505859375 disk.sda2.out‚0$ memory_percent‚96.58446311950684# proc.memory.rssMB‚ 2814.56640625 cpu‚25.18354422769106 disk.drbd1260.in‚0®Ä4‡œÅ‚(:‘ ¤ÞêÔ¸¢Ñù gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚127.841' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.568199999999997 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0òn2Æ- _wandbruntime‚1523  _runtime‚1523õ1ù{ÂǼ _step‚68 % lr-AdamW/pg1‚0.000683270563175077 % lr-AdamW/pg2‚0.000683270563175077 trainer/global_step‚899 $ _timestamp‚1.788522285398815e+09 _runtime‚1523.619040792´ÓVîÆÈÀ % lr-AdamW/pg1‚0.000683270563175077 % lr-AdamW/pg2‚0.000683270563175077 trainer/global_step‚899 $ _timestamp‚1.788522285398815e+09 _runtime‚1523.619040792 _step‚68D›&2É- _wandbruntime‚1523  _runtime‚1523c ùÁ ʼO PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚24096530 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚24071684 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15995494 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 329989632 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚35552724769372 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9730520 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6264292 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚35552970782268 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚35375167098880 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚35375167098880 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚37142228726272 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚37532672504320 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 _step‚69 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1767043624448 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1799802561536 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚37141977821696 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚35375185101824 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1766810722816 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚37141977821696 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16459991 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 306937856 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15994810 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1761814145932 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 _runtime‚1523.620979036 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1767043624448 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16459989 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚35375185101824 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚16459989 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚40531673 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16459991 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 302147072 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 65619886086Ê;aÆ'ËÀO OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚39332499005952 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚24071684 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚33790910623440 % _timestamp‚1.7885222854008214e+09 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚37141977821696 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16459991 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 306937856 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15994810 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1761814145932 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 _runtime‚1523.620979036 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1767043624448 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16459989 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚35375185101824 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚16459989 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚40531673 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16459991 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 302147072 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚24096530 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚24071684 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15995494 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 329989632 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚35552724769372 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9730520 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6264292 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚35552970782268 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚35375167098880 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚35375167098880 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚37142228726272 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚37532672504320 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 _step‚69 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1767043624448 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1799802561536 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚37141977821696 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚35375185101824 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1766810722816 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9741320 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚39376188006912 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16478266 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1801522510848 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚35592213615782 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16013096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1801807592448 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚35592459628886 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1764001608662 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1768774748672 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9741322 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6271774 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚37183478802432 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚675 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚673 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚37574380414464 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 _runtime‚1524.621716475 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚37574356474368 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302147072 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16012421 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 285081600 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1768774748672 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚33828440980864 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚24098376 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚39375878985216 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 % _timestamp‚1.7885222864000716e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16478266 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚33828458020224 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚24123248 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚35414453135872 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 309021696 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚40601514 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚37183227884544 _step‚70 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16478264 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚37183478802432 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚35414471138816 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚40601514 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6271101 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚24123248 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚24098376 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚40576640 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 329989632 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚35414471138816 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1769007663616 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16478264 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1763772634918ÙbºÐ&ÎÊM ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9741322 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6271774 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚37183478802432 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚675 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚673 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚37574380414464 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 _runtime‚1524.621716475 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚37574356474368 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 302147072 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16012421 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 285081600 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1768774748672 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚33828440980864 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚24098376 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚39375878985216 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 % _timestamp‚1.7885222864000716e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16478266 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚33828458020224 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚24123248 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚35414453135872 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 309021696 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚40601514 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚37183227884544 _step‚70 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16478264 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚37183478802432 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚35414471138816 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚40601514 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6271101 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚24123248 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚24098376 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚40576640 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 329989632 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚35414471138816 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1769007663616 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16478264 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1763772634918 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1769007663616 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9741320 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚39376188006912 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16478266 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1801522510848 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚35592213615782 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16013096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1801807592448 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚35592459628886 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1764001608662 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1768774748672 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300FωÌ2Ï- _wandbruntime‚1524  _runtime‚1524:ù?ÐÊ 1loss/FBt4/assignment_loss‚0.22030668886477275 /loss/FRt2/detection_loss‚2.2887454146146773 _runtime‚ 1524.62337387 0loss/FBt3/assignment_loss‚0.7989197969436646 0loss/SRqqt2/detection_loss‚2.628289793476914 /loss/FRt2/assignment_loss‚6.150922966003418 1loss/SRqqt1/assignment_loss‚2.679776688416799 1loss/FBt2/assignment_loss‚0.44004294290898543 .loss/FRt1/detection_loss‚2.539571279287338 1loss/SRqqt1/detection_loss‚2.5781762152910233 0loss/SRqqt4/detection_loss‚4.364777485529582 %loss/total_loss‚64.77371583021528 epoch‚0 2loss/SRqqt2/assignment_loss‚2.9970364462245596 1loss/SRqqt3/assignment_loss‚2.605864302802632 0loss/FBt1/assignment_loss‚0.6629780896666737 .loss/FRt4/detection_loss‚2.8132$Ò %H27586746216 1loss/SRqqt4/assignment_loss‚2.475174307823181 1loss/SRqqt3/detection_loss‚0.9100105611763838 /loss/FBt1/detection_loss‚3.4701609123836863 /loss/FBt2/detection_loss‚1.9241722106933623 trainer/global_step‚899 .loss/FBt3/detection_loss‚5.457909345626831 _step‚71 /loss/FRt1/assignment_loss‚6.245012474060059 /loss/FRt3/assignment_loss‚4.097844673239666 .loss/FRt4/assignment_loss‚2.80016996383667 /loss/FRt3/detection_loss‚2.5803231065687924 % _timestamp‚1.7885222864027033e+09 .loss/FBt4/detection_loss‚1.044302578029387€´sLTÑÎ .loss/FRt4/detection_loss‚2.813227586746216 1loss/SRqqt4/assignment_loss‚2.475174307823181 1loss/SRqqt3/detection_loss‚0.9100105611763838 /loss/FBt1/detection_loss‚3.4701609123836863 /loss/FBt2/detection_loss‚1.9241722106933623 trainer/global_step‚899 .loss/FBt3/detection_loss‚5.457909345626831 _step‚71 /loss/FRt1/assignment_loss‚6.245012474060059 /loss/FRt3/assignment_loss‚4.097844673239666 .loss/FRt4/assignment_loss‚2.80016996383667 /loss/FRt3/detection_loss‚2.5803231065687924 % _timestamp‚1.7885222864027033e+09 .loss/FBt4/detection_loss‚1.044302578029387 1loss/FBt4/assignment_loss‚0.22030668886477275 /loss/FRt2/detection_loss‚2.2887454146146773 _runtime‚ 1524.62337387 0loss/FBt3/assignment_loss‚0.7989197969436646 0loss/SRqqt2/detection_loss‚2.628289793476914 /loss/FRt2/assignment_loss‚6.150922966003418 1loss/SRqqt1/assignment_loss‚2.679776688416799 1loss/FBt2/assignment_loss‚0.44004294290898543 .loss/FRt1/detection_loss‚2.539571279287338 1loss/SRqqt1/detection_loss‚2.5781762152910233 0loss/SRqqt4/detection_loss‚4.364777485529582 %loss/total_loss‚64.77371583021528 epoch‚0 2loss/SRqqt2/assignment_loss‚2.9970364462245596 1loss/SRqqt3/assignment_loss‚2.605864302802632 0loss/FBt1/assignment_loss‚0.6629780896666737GGeÏËvÒ‚(:ë ³ÞêÔ´‡ø network.recv‚545689) disk./.usagePercent‚78.90082596837205' disk.nvme0c0n1.out‚27722.4755859375 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda.in‚0( proc.memory.availableMB‚ 559.50390625 proc.cpu.threads‚17 network.sent‚ 1.457588e+06! disk.nvme0n1.in‚ 3467.77734375% disk.nvme0n1.out‚27722.4755859375 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 44.41015625" proc.memory.rssMB‚ 2814.5703125% disk./.usageGB‚1175.6793251037598 disk.dm-1.out‚ 44.41015625 disk.drbd1260.out‚ 0.2109375 disk.sda2.out‚0 disk.dm-0.in‚ 3467.77734375 disk.sda.out‚ 44.41015625* proc.memory.percent‚17.178773880004883 cpu‚25.182799769199125# disk.nvme0c0n1.in‚ 3467.77734375 disk.dm-1.in‚0" disk.dm-0.out‚27722.4755859375$ memory_percent‚96.58505916595459ʧ -œÓ‚(:‘ ³ÞêÔýª¦ø gpu.0.gpu‚16.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚106.292' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.258399999999998 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸µ>‘Ô‚(:† ÂÞêÔ †÷õ gpu.0.gpu‚21.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚29.0 gpu.0.powerWatts‚101.848' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.3696 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ñ“yÕ‚(:î ÂÞêÔÔàèø* proc.memory.percent‚17.178773880004883) disk./.usagePercent‚78.90082468833438 disk.sda2.in‚0 disk.sda2.out‚0# disk.dm-0.out‚27891.23583984375# disk.nvme0c0n1.in‚ 3574.47265625! disk.nvme0n1.in‚ 3574.47265625& disk.nvme0n1.out‚27891.23583984375$ memory_percent‚96.58503532409668 network.sent‚ 1.506383e+06 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 45.03515625 disk.drbd1260.in‚0 disk.sda.in‚0 cpu‚25.184658868862773 proc.cpu.threads‚17 network.recv‚550466 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.21484375 disk.sda.out‚ 45.03515625 disk.sda3.in‚0' proc.memory.availableMB‚ 559.5078125% disk./.usageGB‚1175.6793060302734 disk.dm-0.in‚ 3574.47265625 disk.sda3.out‚ 45.03515625( disk.nvme0c0n1.out‚27891.23583984375" proc.memory.rssMB‚ 2814.5703125Ù¹9ΛÖ‚(: ÑÞêÔëâ†÷ gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚126.26' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.252000000000002 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0V`?Æyׂ(:î ÑÞêÔ‡­³ú network.recv‚554790) disk./.usagePercent‚78.89410884268867 disk.sda3.in‚0! disk.nvme0n1.in‚ 3574.47265625& disk.nvme0n1.out‚28299.72802734375 disk.dm-1.out‚ 45.61328125 disk.drbd1260.in‚0 disk.dm-0.in‚ 3574.47265625 network.sent‚ 1.512288e+06% disk./.usageGB‚1175.5792350769043 disk.sda3.out‚ 45.61328125# disk.nvme0c0n1.in‚ 3574.47265625 disk.sda1.in‚0 disk.sda.in‚0( proc.memory.availableMB‚ 559.00390625* proc.memory.percent‚17.178773880004883 disk.sda1.out‚0 disk.dm-1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0# disk.dm-0.out‚28299.72802734375 disk.sda.out‚ 45.61328125$ memory_percent‚96.58811092376709 cpu‚25.18490563756426( disk.nvme0c0n1.out‚28299.72802734375 disk.drbd1260.out‚ 0.21484375" proc.memory.rssMB‚ 2814.5703125 proc.cpu.threads‚17¾c¥Ë‘Ø‚(:† àÞêÔ¦Ó¹ö gpu.0.gpu‚12.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚117.414' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.4828 gpu.0.smClock‚1305 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0JŸ\qÙ‚(:æ àÞêÔÝä×ø disk.sda.in‚0$ memory_percent‚96.58510684967041( proc.memory.availableMB‚ 559.49609375" proc.memory.rssMB‚ 2814.5703125 disk.dm-1.out‚ 45.9140625 network.sent‚ 1.518522e+06! disk.nvme0n1.in‚ 3574.47265625 disk.sda1.out‚0 disk.dm-1.in‚0 disk.sda2.in‚0* proc.memory.percent‚17.178773880004883 proc.cpu.threads‚17# disk.nvme0c0n1.in‚ 3574.47265625 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 45.9140625 cpu‚25.183869534402877 disk.drbd1260.out‚0.21875) disk./.usagePercent‚78.89938899808236% disk./.usageGB‚1175.6579132080078' disk.nvme0c0n1.out‚28573.1884765625% disk.nvme0n1.out‚28573.1884765625 disk.sda.out‚ 45.9140625 disk.sda3.in‚0 network.recv‚559471 disk.drbd1260.in‚0 disk.dm-0.in‚ 3574.47265625" disk.dm-0.out‚28573.1884765625Q­x›Ú‚(: ïÞêÔêò ø gpu.0.gpu‚12.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚29.0 gpu.0.powerWatts‚92.527' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚18.505399999999998 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Â9zsÛ‚(:è ïÞêÔ‡žü disk.sda.out‚ 46.26953125! disk.nvme0n1.in‚ 3574.47265625 disk.sda2.in‚0 disk.dm-0.in‚ 3574.47265625 disk.sda3.in‚0 disk.sda3.out‚ 46.26953125 disk.dm-1.in‚0 disk.dm-1.out‚ 46.26953125 disk.drbd1260.out‚0.21875$ memory_percent‚96.58515453338623( proc.memory.availableMB‚ 559.48828125( disk./.usagePercent‚78.9046752976569% disk./.usageGB‚1175.7366828918457# disk.nvme0c0n1.in‚ 3574.47265625 disk.sda2.out‚0* proc.memory.percent‚17.178773880004883 cpu‚25.184208818370205' disk.nvme0c0n1.out‚29037.2861328125% disk.nvme0n1.out‚29037.2861328125 disk.sda1.out‚0 disk.drbd1260.in‚0" disk.dm-0.out‚29037.2861328125" proc.memory.rssMB‚ 2814.5703125 proc.cpu.threads‚17 disk.sda1.in‚0 network.sent‚ 1.523666e+06 network.recv‚562686 disk.sda.in‚0èú•2Ü- _wandbruntime‚1597  _runtime‚1597gà»ÂÄݾ & lr-AdamW/pg2‚0.0007212722630179622 trainer/global_step‚949 % _timestamp‚1.7885223591149855e+09 _runtime‚ 1597.33735095 _step‚72 & lr-AdamW/pg1‚0.0007212722630179622ee“ÈÞ _runtime‚ 1597.33735095 _step‚72 & lr-AdamW/pg1‚0.0007212722630179622 & lr-AdamW/pg2‚0.0007212722630179622 trainer/global_step‚949 % _timestamp‚1.7885223591149855e+09HœŽß2ß- _wandbruntime‚1597  _runtime‚1597›Ù—GÄ'à¾O O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚37339468948480 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚41516435633664 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚39616894946816 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1865259331072 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚42808223 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚39616871006720 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚25434482 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6612320 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚35667445534000 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚25409636 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚697 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚39204746282496 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17373741 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚41516755154432 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10270618 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚25434482 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚699 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17373741 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1859981159976 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚39204495377920 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚17373739 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10270620 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚1597.337761733 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚35667428494640 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 % _timestamp‚1.7885223591172912e+09 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1865026429440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1865259331072 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16882241 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚37339468948480 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚39204495377920 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16882940 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1865026429440 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 trainer/global_step‚949 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚17373739N$TuÈ'áÂO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚25409636 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚697 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚39204746282496 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17373741 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚41516755154432 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10270618 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚25434482 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚699 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17373741 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1859981159976 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚39204495377920 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚17373739 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10270620 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚1597.337761733 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚35667428494640 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 % _timestamp‚1.7885223591172912e+09 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1865026429440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1865259331072 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16882241 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚37339468948480 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚39204495377920 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16882940 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1865026429440 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 trainer/global_step‚949 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚17373739 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚37339468948480 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚41516435633664 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚39616894946816 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1865259331072 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚42808223 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚39616871006720 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚25434482 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6612320 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚35667445534000 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848I¸}3-2â- _wandbruntime‚1599  _runtime‚1599vç\ê ãÈM ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚37378772988416 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16900329 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 298535936 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1901568016384 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚35704958852064 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚25461200 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚37378754985472 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚25436328 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚686 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚39658578916864 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1861939649170 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚37566915540594 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1866990455296 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚39245745440768 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.c<|Oäurrent‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚42828342 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17392016 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1866990455296 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10281420 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚1599.052609769 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17392014 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚25461200 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚37378754985472 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 274595840 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚39658554976768 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1901842612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 308438528 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚37566669527490 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚42853216 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1861710675426 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚37378772988416 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6618907 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16899641 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚41560421529088 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 trainer/global_step‚949 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25436328 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508”6FÒ&äÌM O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 % _timestamp‚1.7885223608300045e+09 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚25461200 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚37378754985472 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 274595840 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚39658554976768 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1901842612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 308438528 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚37566669527490 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚42853216 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1861710675426 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚37378772988416 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6618907 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16899641 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚41560421529088 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 trainer/global_step‚949 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25436328 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚37378772988416 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16900329 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 298535936 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1901568016384 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚35704958852064 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚25461200 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚37378754985472 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚25436328 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚686 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚39658578916864 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1861939649170 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚37566915540594 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1866990455296 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚39245745440768 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚42828342 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17392016 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1866990455296 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10281420 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚1599.052609769 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17392014 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 ODeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1963242136064 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚26747588 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 _runtime‚1683.146285712 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17769654 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚41701093449216 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1999974906368 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10810720 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 308438528 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚26772434 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚39303788801024 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6958936 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚43701068355584 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 315326464 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚41701069509120 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚37543963405200 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚703 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚43700753029120 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚41267263838720 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1963475037696 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚26747588 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 291386368 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1999683520000 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚45035077 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚37543946365840 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1963242136064 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚41267012934144 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚41267012934144 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1963475037696 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17770357 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1957690226948 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚39303770798080 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚41267263838720 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909^È'ùÂO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚703 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚43700753029120 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚41267263838720 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1963475037696 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚26747588 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 291386368 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1999683520000 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚45035077 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚37543946365840 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1963242136064 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚41267012934144 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚41267012934144 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1963475037696 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17770357 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1957690226948 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚39303770798080 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚41267263838720 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18287491 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 % _timestamp‚1.7885224449267116e+09 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚18287489 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1963242136064 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚26747588 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 _runtime‚1683.146285712 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17769654 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚41701093449216 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1999974906368 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10810720 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 308438528 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚26772434 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚39303788801024 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6958936 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚43701068355584 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 315326464 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚41701069509120 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚37543963405200M ta2ú- _wandbruntime‚1684  _runtime‚1684Ÿ©¶Î&ûÈM VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚41308513914880 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚41742753479168 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 _runtime‚1684.060172231 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚37581476723264 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18305764 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1965439076864 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚26774280 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17787685 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336281088 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚39343056835072 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚676 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚39541125439198 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2001964519936 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 287178752 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 % _timestamp‚1.7885224458399134e+09 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚39541371452302 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚37581493762624 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚41308513914880 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1965439076864 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17788361 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚41308262996992 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚45104918 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 311118848 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 trainer/global_step‚999 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18305766 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1965206161920 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚45080044 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚43744430820352 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚45080044 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 v,*\üÌM XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚37581493762624 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚41308513914880 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1965439076864 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17788361 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚41308262996992 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚45104918 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 311118848 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 trainer/global_step‚999 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18305766 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1965206161920 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚45080044 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚43744430820352 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚45080044 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚41308513914880 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚41742753479168 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 _runtime‚1684.060172231 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚37581476723264 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18305764 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1965439076864 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚26774280 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17787685 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336281088 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚39343056835072 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚676 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚39541125439198 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2001964519936 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 287178752 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 % _timestamp‚1.7885224458399134e+09 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚39541371452302NöÖ{2ý- _wandbruntime‚1684  _runtime‚1684}šÐŽKþÅ -loss/FRt1/detection_loss‚2.36464368502299 1loss/SRqqt3/detection_loss‚0.8036314811941017 %loss/total_loss‚68.30568333822728 /loss/FRt1/assignment_loss‚6.060742537180583 /loss/FRt3/assignment_loss‚4.546562823696413 /loss/SRqqt2/assignment_loss‚3.1427201451482 2loss/SRqqt4/assignment_loss‚3.8267009258270264 /loss/FBt3/assignment_loss‚0.920953372376971 -loss/FRt4/detection_loss‚2.38338402623222 1loss/SRqqt2/detection_loss‚2.3384521128358067 .loss/FBt3/detection_loss‚5.358704566955566 0loss/FBt1/assignment_loss‚0.7702429518152052 .loss/FRt2/detection_loss‚4.328149310180119 0loss/SRqqt4/detection_loss‚4.735318623483181 epoch‚0 0loss/FBt4/assignment_loss‚0.2835652440248375 /loss/FBt4/detection_loss‚0.8991382533628771 _runtime‚1684.061802116 _step‚79 .loss/FBt1/detection_loss‚2.348256754023688 /loss/FBt2/detection_loss‚1.5076127030121784 2loss/SRqqt1/assignment_loss‚2.5869181056817374 0loss/SRqqt1/detection_loss‚2.748555781940619 /loss/FBt2/assignment_loss‚0.470001290291649 /loss/FRt3/detection_loss‚1.8155036110808884 /loss/FRt4/assignment_loss‚4.441988683882213 trainer/global_step‚999 % _timestamp‚1.7885224458424926e+09 /loss/FRt2/assignment_loss‚6.986349582672119 2loss/SRqqt3/assignment_loss‚2.6375867663060917ú±uOÿÉ /loss/FRt4/assignment_loss‚4.441988683882213 trainer/global_step‚999 % _timestamp‚1.7885224458424926e+09 /loss/FRt2/assignment_loss‚6.986349582672119 2loss/SRqqt3/assignment_loss‚2.6375867663060917 -loss/FRt1/detection_loss‚2.36464368502299 1loss/SRqqt3/detection_loss‚0.8036314811941017 %loss/total_loss‚68.30568333822728 /loss/FRt1/assignment_loss‚6.060742537180583 /loss/FRt3/assignment_loss‚4.546562823696413 /loss/SRqqt2/assignment_loss‚3.1427201451482 2loss/SRqqt4/assignment_loss‚3.8267009258270264 /loss/FBt3/assignment_loss‚0.920953372376971 -loss/FRt4/detection_loss‚2.38338402623222 1loss/SRqqt2/detection_loss‚2.3384521128358067 .loss/FBt3/detection_loss‚5.358704566955566 0loss/FBt1/assignment_loss‚0.7702429518152052 .loss/FRt2/detection_loss‚4.328149310180119 0loss/SRqqt4/detection_loss‚4.735318623483181 epoch‚0 0loss/FBt4/assignment_loss‚0.2835652440248375 /loss/FBt4/detection_loss‚0.8991382533628771 _runtime‚1684.061802116 _step‚79 .loss/FBt1/detection_loss‚2.348256754023688 /loss/FBt2/detection_loss‚1.5076127030121784 2loss/SRqqt1/assignment_loss‚2.5869181056817374 0loss/SRqqt1/detection_loss‚2.748555781940619 /loss/FBt2/assignment_loss‚0.470001290291649 /loss/FRt3/detection_loss‚1.8155036110808884O(7‡€‚(:„ ØßêÔ ÓÑ÷ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚29.0 gpu.0.powerWatts‚92.744' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.5488 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÍTà^‚(:Ó ØßêÔÆþø network.recv‚592603 disk.sda.out‚49.1875 disk.sda3.in‚0$ memory_percent‚96.58880233764648* proc.memory.percent‚17.178797721862793 cpu‚25.185828396979424 proc.cpu.threads‚17 disk.sda3.out‚49.1875 disk.nvme0c0n1.in‚3896.875 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.23828125 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚3896.875) disk./.usagePercent‚78.92015556123732% disk./.usageGB‚1175.9673500061035 disk.sda.in‚0( disk.nvme0c0n1.out‚30749.81787109375& disk.nvme0n1.out‚30749.81787109375 disk.sda1.out‚0# disk.dm-0.out‚30749.81787109375& proc.memory.availableMB‚ 558.890625 network.sent‚ 1.607153e+06 disk.nvme0n1.in‚3896.875 disk.sda1.in‚0 disk.dm-1.out‚49.1875# proc.memory.rssMB‚ 2814.57421875Bd‚‚(:… çßêÔíù‡÷ gpu.0.gpu‚13.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚29.0 gpu.0.powerWatts‚92.852' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.5704 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0§ì™fƒ‚(:Û çßêÔº´ýø( disk.nvme0c0n1.out‚31157.22412109375 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.sda.in‚0$ memory_percent‚96.58880233764648# proc.memory.rssMB‚ 2814.57421875* proc.memory.percent‚17.178797721862793 network.sent‚ 1.655874e+06) disk./.usagePercent‚78.92173179962565$ disk./.usageGB‚1175.990837097168 disk.sda1.in‚0 disk.dm-1.in‚0 disk.dm-0.in‚3896.875 disk.sda3.in‚0 disk.sda3.out‚ 49.4921875 disk.nvme0n1.in‚3896.875& disk.nvme0n1.out‚31157.22412109375 disk.drbd1260.out‚ 0.23828125 disk.sda2.in‚0# disk.dm-0.out‚31157.22412109375 disk.sda.out‚ 49.4921875 network.recv‚596907 disk.dm-1.out‚ 49.4921875 disk.sda2.out‚0& proc.memory.availableMB‚ 558.890625 cpu‚25.185097792074266 proc.cpu.threads‚17 disk.nvme0c0n1.in‚3896.875ÜÁ»M‘„‚(:† ößêÔ‰˜éõ gpu.0.gpu‚25.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚106.541' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.3082 gpu.0.smClock‚1305 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ×°i…‚(:Þ ößêÔâòá÷ disk.drbd1260.in‚0 disk.sda2.out‚0$ memory_percent‚96.58732414245605 cpu‚25.18441455242861 disk.sda3.in‚0 disk.sda3.out‚ 49.796875& disk.nvme0c0n1.out‚31383.998046875 disk.nvme0n1.in‚ 4031.0859375 disk.sda1.out‚0' proc.memory.availableMB‚ 559.1328125 proc.cpu.threads‚17) disk./.usagePercent‚78.93700162501295 disk.sda.out‚ 49.796875" disk.nvme0c0n1.in‚ 4031.0859375$ disk.nvme0n1.out‚31383.998046875 disk.dm-1.in‚0 disk.dm-1.out‚ 49.796875# proc.memory.rssMB‚ 2814.57421875* proc.memory.percent‚17.178797721862793 network.sent‚ 1.660023e+06% disk./.usageGB‚1176.2183685302734 disk.drbd1260.out‚ 0.2421875 disk.sda2.in‚0 disk.dm-0.in‚ 4031.0859375! disk.dm-0.out‚31383.998046875 disk.sda.in‚0 disk.sda1.in‚0 network.recv‚600101lâóÕœ†‚(:‘ …àêÔÇ©üö gpu.0.gpu‚25.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚111.896' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.379199999999997 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0v¬‚,l‡‚(:á …àêÔÈÈïú cpu‚25.184036017209912) disk./.usagePercent‚78.93850822935194" disk.nvme0c0n1.in‚ 4031.0859375 disk.sda1.out‚0 disk.dm-1.out‚ 50.26953125 disk.sda2.out‚0 disk.sda.in‚0 network.recv‚604694& disk.nvme0c0n1.out‚31825.294921875 disk.sda1.in‚0 disk.drbd1260.in‚0 disk.sda2.in‚0# proc.memory.availableMB‚559.125# proc.memory.rssMB‚ 2814.57421875 proc.cpu.threads‚17% disk./.usageGB‚1176.2408180236816 disk.nvme0n1.in‚ 4031.0859375$ disk.nvme0n1.out‚31825.294921875 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.2421875! disk.dm-0.out‚31825.294921875 disk.sda3.out‚ 50.26953125 network.sent‚ 1.665972e+06 disk.dm-0.in‚ 4031.0859375 disk.sda.out‚ 50.26953125 disk.sda3.in‚0$ memory_percent‚96.58737182617188* proc.memory.percent‚17.178797721862793p˜¤Óœˆ‚(:‘ ”àêÔ̱Åö gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚136.871' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.374200000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0\¨A~i‰‚(:Þ ”àêÔëÎÇ÷ network.sent‚ 1.672037e+06 network.recv‚609349 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.24609375 disk.sda2.in‚0 disk.sda.out‚50.71875 disk.sda3.in‚0 disk.nvme0n1.in‚ 4031.0859375% disk./.usageGB‚1176.2408180236816 disk.dm-1.out‚50.71875 disk.sda2.out‚0& disk.nvme0n1.out‚31931.72998046875 disk.sda1.out‚0$ memory_percent‚96.58737182617188* proc.memory.percent‚17.178797721862793 proc.cpu.threads‚17) disk./.usagePercent‚78.93850822935194 disk.drbd1260.in‚0 disk.sda.in‚0" disk.nvme0c0n1.in‚ 4031.0859375( disk.nvme0c0n1.out‚31931.72998046875# proc.memory.availableMB‚559.125# proc.memory.rssMB‚ 2814.57421875 cpu‚25.18357200052645 disk.dm-0.in‚ 4031.0859375# disk.dm-0.out‚31931.72998046875 disk.sda3.out‚50.71875 disk.sda1.in‚0Z"wŠ‚(:„ £àêÔ¡³Í÷ gpu.0.gpu‚4.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚29.0 gpu.0.powerWatts‚95.327' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.0654 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µ~ût‹‚(:é £àêԕجù# disk.dm-0.out‚32080.56982421875 disk.sda3.in‚0' proc.memory.availableMB‚ 559.1171875# proc.memory.rssMB‚ 2814.57421875 disk.dm-0.in‚ 4031.0859375 disk.sda.in‚0( disk.nvme0c0n1.out‚32080.56982421875 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.24609375 disk.sda2.in‚0 disk.sda2.out‚0* proc.memory.percent‚17.178797721862793) disk./.usagePercent‚78.93850950938962$ disk./.usageGB‚1176.240837097168 disk.sda.out‚ 51.26171875 disk.sda3.out‚ 51.26171875& disk.nvme0n1.out‚32080.56982421875 disk.sda1.out‚0 disk.dm-1.out‚ 51.26171875 proc.cpu.threads‚17 network.recv‚612684" disk.nvme0c0n1.in‚ 4031.0859375 disk.nvme0n1.in‚ 4031.0859375 disk.sda1.in‚0 disk.dm-1.in‚0# memory_percent‚96.5874195098877 cpu‚25.18310717818883 network.sent‚ 1.677003e+06&g®[2Œ- _wandbruntime‚1770  _runtime‚1770îkÅ¿ & lr-AdamW/pg1‚0.0007972756627037328 & lr-AdamW/pg2‚0.0007972756627037328 trainer/global_step‚1049 % _timestamp‚1.7885225320468087e+09 _runtime‚ 1770.26844725 _step‚80_ðO ÉŽÃ trainer/global_step‚1049 % _timestamp‚1.7885225320468087e+09 _runtime‚ 1770.26844725 _step‚80 & lr-AdamW/pg1‚0.0007972756627037328 & lr-AdamW/pg2‚0.0007972756627037328PR72- _wandbruntime‚1770  _runtime‚1770ÖLƒÅ'¿O H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2061457842688 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 308438528 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚41476338517392 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11350818 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚19201241 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11350820 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚28085540 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚28085540 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2061690744320 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 trainer/global_step‚1049 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚39420464237040 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚1770.272028892 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18659248 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚711 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7307717 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚39420481276400 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2099577829376 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2099867118592 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚41268072647680 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚713 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚43329781394944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 289289216 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚81 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19201239 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚47286779 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 313229312 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚43785291951616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚43329530490368 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚41268090650624 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2061690744320 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚43785268011520 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚41476092504496 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2061457842688 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19201239 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚45884845840896 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚28110386 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336281088 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18659248 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚711 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7307717 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚39420481276400 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2099577829376 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966133760 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2099867118592 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚41268072647680 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚713 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚43329781394944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 289289216 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚81 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19201239 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚47286779 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 313229312 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚43785291951616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚43329530490368 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚41268090650624 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2061690744320 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚43785268011520 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚41476092504496 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2061457842688 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19201239 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚45884845840896 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚28110386 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336281088 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2061457842688 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 308438528 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚41476338517392 ?fnÄüI9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11350818 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚19201241 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11350820 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚28085540 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚28085540 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920Q /Ó2’- _wandbruntime‚1772  _runtime‚1772¤\êsÏ&“ÉM REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19219516 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7314300 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 308438528 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚28137104 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚678 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚45928525614592 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚47356620 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚47356620 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚28112232 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336281088 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚41307376687616 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18676598 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7314976 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2101858714624 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19219516 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚47331746 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2063654783488 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11361622 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2101573633024 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚43371031471104 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2063654783488 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚41515827364010 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18675920 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚45928834636288 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚43826975921664 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚1049 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚39457994594464 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚47331746 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚28112232 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 285081600 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚41307376687616 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚676 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2057586756442 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19219514 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28137104 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2063421868544 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚43371031471104 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2063421868544 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309¢×¼Ó&”ÍM REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19219516 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7314300 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 308438528 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚28137104 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚678 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚45928525614592 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚47356620 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚47356620 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚28112232 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336281088 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚41307376687616 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18676598 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7314976 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2101858714624 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19219516 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚47331746 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2063654783488 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11361622 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2101573633024 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚43371031471104 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2063654783488 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚41515827364010 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18675920 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚45928834636288 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚43826975921664 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚1049 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚39457994594464 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚47331746 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30751 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚28112232 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 285081600 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚41307376687616 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚676 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2057586756442 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19219514 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28137104 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2063421868544 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚43371031471104 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2063421868544 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309RöînÉ2•- _wandbruntime‚1772  _runtime‚1772ÔJBRB–¼ 0loss/FRt3/assignment_loss‚4.5344603061676025 .loss/SRqqt2/detection_loss‚2.4465079108874 % _timestamp‚1.7885225343502505e+09 .loss/FRt1/assignment_loss‚5.71726900491959 /loss/FBt2/detection_loss‚3.6635947465896606 epoch‚0 0loss/FBt2/assignment_loss‚0.8006853818893432 /loss/FRt3/detection_loss‚1.8726929924317768 .loss/FBt4/detection_loss‚1.018626892985776 %loss/total_loss‚62.05083183762875 _step‚83 0loss/FRt2/assignment_loss‚5.6236701011657715 1loss/SRqqt3/assignment_loss‚2.490780277769695 2loss/SRqqt4/assignment_loss‚3.0833880404631295 loss/FBt3/assignment_loss‚0 .loss/FRt1/detection_loss‚2.443655331929525 1loss/SRqqt3/detection_loss‚0.9446168548261472 /loss/FBt1/detection_loss‚2.7997244000434875 /loss/FRt4/assignment_loss‚3.825016525658694 2loss/SRqqt1/assignment_loss‚2.2904678082466123 /loss/FBt3/detection_loss‚3.8878121972084045 2loss/SRqqt2/assignment_loss‚2.4812117417653403 0loss/FBt1/assignment_loss‚0.5263960175202129 .loss/FRt2/detection_loss‚3.051141829326235 1loss/FBt4/assignment_loss‚0.22380804252491088 /loss/FRt4/detection_loss‚2.0564054386182264 0loss/SRqqt4/detection_loss‚3.444053371747335 trainer/global_step‚1049 1loss/SRqqt1/detection_loss‚2.8248466229438782 _runtime‚ 1772.56968762€s%]F—À 2loss/SRqqt2/assignment_loss‚2.4812117417653403 0loss/FBt1/assignment_loss‚0.5263960175202129 .loss/FRt2/detection_loss‚3.051141829326235 1loss/FBt4/assignment_loss‚0.22380804252491088 /loss/FRt4/detection_loss‚2.0564054386182264 0loss/SRqqt4/detection_loss‚3.444053371747335 trainer/global_step‚1049 1loss/SRqqt1/detection_loss‚2.8248466229438782 _runtime‚ 1772.56968762 0loss/FRt3/assignment_loss‚4.5344603061676025 .loss/SRqqt2/detection_loss‚2.4465079108874 % _timestamp‚1.7885225343502505e+09 .loss/FRt1/assignment_loss‚5.71726900491959 /loss/FBt2/detection_loss‚3.6635947465896606 epoch‚0 0loss/FBt2/assignment_loss‚0.8006853818893432 /loss/FRt3/detection_loss‚1.8726929924317768 .loss/FBt4/detection_loss‚1.018626892985776 %loss/total_loss‚62.05083183762875 _step‚83 0loss/FRt2/assignment_loss‚5.6236701011657715 1loss/SRqqt3/assignment_loss‚2.490780277769695 2loss/SRqqt4/assignment_loss‚3.0833880404631295 loss/FBt3/assignment_loss‚0 .loss/FRt1/detection_loss‚2.443655331929525 1loss/SRqqt3/detection_loss‚0.9446168548261472 /loss/FBt1/detection_loss‚2.7997244000434875 /loss/FRt4/assignment_loss‚3.825016525658694 2loss/SRqqt1/assignment_loss‚2.2904678082466123 /loss/FBt3/detection_loss‚3.8878121972084045S½A¦˜‚(:„ ²àêÔ¨­æ÷ gpu.0.gpu‚8.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚102.135' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.427 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0½5\™‚(:Ñ ²àêÔ¢äëø network.sent‚ 1.682307e+06 network.recv‚616325 disk.sda1.out‚0 disk.dm-0.out‚ 32431.6015625 disk.sda.in‚0 disk.sda.out‚51.53125 disk.sda3.in‚0" disk.nvme0c0n1.in‚ 4180.3671875 disk.dm-1.in‚0 disk.dm-1.out‚51.53125 disk.drbd1260.in‚0 disk.sda2.out‚0 disk.sda3.out‚51.53125$ disk.nvme0c0n1.out‚ 32431.6015625 disk.nvme0n1.in‚ 4180.3671875" disk.nvme0n1.out‚ 32431.6015625% disk./.usageGB‚1176.2173500061035 disk.sda1.in‚0 disk.drbd1260.out‚0.25( proc.memory.availableMB‚ 559.05859375* proc.memory.percent‚17.178797721862793 cpu‚25.182578413073614 proc.cpu.threads‚17( disk./.usagePercent‚78.9369332710013 disk.sda2.in‚0 disk.dm-0.in‚ 4180.3671875$ memory_percent‚96.58777713775635# proc.memory.rssMB‚ 2814.57421875ô_—cš‚(:„ ÁàêÔê·ö gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚99.967' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.9934 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¨€G²h›‚(:Ý ÁàêÔÂòïø* proc.memory.percent‚17.178797721862793 proc.cpu.threads‚17 disk.drbd1260.in‚0 disk.dm-0.in‚ 4180.3671875 disk.sda3.in‚0% disk.nvme0c0n1.out‚32690.69140625 disk.sda1.in‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 2814.57421875 cpu‚25.182178466763453% disk./.usageGB‚1176.7192153930664 disk.drbd1260.out‚0.25 disk.sda2.out‚0 disk.sda.in‚0 disk.sda.out‚ 51.94921875 disk.nvme0n1.in‚ 4180.3671875 disk.dm-1.in‚0$ memory_percent‚96.58787250518799 network.sent‚ 1.733672e+06 network.recv‚621660 disk.sda3.out‚ 51.94921875( proc.memory.availableMB‚ 559.04296875( disk./.usagePercent‚78.9706138782135 disk.sda2.in‚0 disk.dm-0.out‚32690.69140625" disk.nvme0c0n1.in‚ 4180.3671875# disk.nvme0n1.out‚32690.69140625 disk.dm-1.out‚ 51.94921875[ig‘œ‚(:† ÐàêԻ¦ö gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚131.803' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.3606 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0üשqr‚(:ç ÐàêÔ䨿ù cpu‚25.18163486610252 proc.cpu.threads‚17 network.sent‚ 1.737952e+06 disk.nvme0n1.in‚ 4180.3671875 disk.dm-0.in‚ 4180.3671875" disk.dm-0.out‚33257.9384765625* proc.memory.percent‚17.178797721862793 disk.sda3.in‚0 disk.sda3.out‚ 52.55859375" disk.nvme0c0n1.in‚ 4180.3671875' disk.nvme0c0n1.out‚33257.9384765625 disk.sda1.in‚0 disk.drbd1260.out‚ 0.25390625 disk.sda2.out‚0# memory_percent‚96.5878963470459) disk./.usagePercent‚78.97047998627308% disk./.usageGB‚1176.7172203063965% disk.nvme0n1.out‚33257.9384765625 disk.sda1.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 52.55859375 disk.sda2.in‚0 disk.sda.out‚ 52.55859375 network.recv‚626342 disk.drbd1260.in‚0 disk.sda.in‚0' proc.memory.availableMB‚ 559.0390625# proc.memory.rssMB‚ 2814.57421875/ZŽ&iž‚(:Þ ßàêÔ‘¦Ú÷ cpu‚25.1810338556754 proc.cpu.threads‚17 network.sent‚ 1.743155e+06$ disk./.usageGB‚1176.306495666504 disk.sda2.out‚0 disk.sda.out‚ 52.765625 disk.sda3.in‚0" disk.nvme0c0n1.in‚ 4180.3671875 disk.nvme0n1.in‚ 4180.3671875% disk.nvme0n1.out‚33406.6923828125) disk./.usagePercent‚78.94291591106897 disk.sda.in‚0' disk.nvme0c0n1.out‚33406.6923828125 disk.sda1.out‚0 disk.dm-1.out‚ 52.765625$ memory_percent‚96.58946990966797# proc.memory.rssMB‚ 2814.57421875 network.recv‚630121 disk.dm-0.in‚ 4180.3671875" disk.dm-0.out‚33406.6923828125 disk.sda3.out‚ 52.765625 disk.sda1.in‚0 disk.dm-1.in‚0% proc.memory.availableMB‚ 558.78125* proc.memory.percent‚17.178797721862793 disk.sda2.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.25390625B\‚\œŸ‚(:‘ ßàêÔø·ø÷ gpu.0.gpu‚18.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚140.424' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.084799999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¦ý·æ‘ ‚(:† îàêÔêüàõ gpu.0.gpu‚16.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚139.933' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.9866 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0›ó÷t¡‚(:é îàêÔœ­µù$ memory_percent‚96.58806324005127( proc.memory.availableMB‚ 559.01171875 network.sent‚ 1.748201e+06) disk./.usagePercent‚78.94133967268064 disk.sda2.in‚0 disk.sda1.in‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 2814.57421875 proc.cpu.threads‚17 disk.sda3.out‚ 53.19140625" disk.nvme0c0n1.in‚ 4180.3671875 disk.sda2.out‚0" disk.dm-0.out‚33661.5361328125 disk.sda3.in‚0 disk.nvme0n1.in‚ 4180.3671875 disk.dm-1.out‚ 53.19140625* proc.memory.percent‚17.178797721862793% disk.nvme0n1.out‚33661.5361328125 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.2578125 cpu‚25.180609877771502' disk.nvme0c0n1.out‚33661.5361328125 network.recv‚634033% disk./.usageGB‚1176.2830085754395 disk.dm-0.in‚ 4180.3671875 disk.sda.in‚0 disk.sda.out‚ 53.19140625Iq 2¢- _wandbruntime‚1852  _runtime‚1852Uav>Æ£À & lr-AdamW/pg2‚0.0008352773625466181 trainer/global_step‚1099 % _timestamp‚1.7885226143772402e+09 _runtime‚1852.597415202 _step‚84 & lr-AdamW/pg1‚0.0008352773625466181®£î[Ê¤Ä _runtime‚1852.597415202 _step‚84 & lr-AdamW/pg1‚0.0008352773625466181 & lr-AdamW/pg2‚0.0008352773625466181 trainer/global_step‚1099 % _timestamp‚1.7885226143772402e+09T²°ëº2¥- _wandbruntime‚1852  _runtime‚18527Îø¬¦¾O O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚43232374497280 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2199781060608 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚1099 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2200070349824 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 308438528 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚45392048046592 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7655151 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚48069247574528 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 289289216 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚43232392500224 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚41296982108240 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚45392048046592 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20114989 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19546071 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20114991 $ _timestamp‚1.788522614378573e+09 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336281088 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚43232392500224 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚665 _step‚85 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚48069560803840 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 313229312 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20114989 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚29448338 _runtime‚1852.598885368 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚43450548416204 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚49563329 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚43450794429100 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚49538481 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19545404 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚41296999147600 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11890918 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚45869490454016 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚48069247574528 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 289289216 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚43232392500224 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚41296982108240 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚45392048046592 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20114989 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19546071 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20114991 $ _timestamp‚1.788522614378573e+09 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336281088 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚43232392500224 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚665 _step‚85 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚48069560803840 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 313229312 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20114989 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚29448338 _runtime‚1852.598885368 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚43450548416204 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚49563329 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚43450794429100 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚49538481 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19545404 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚41296999147600 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11890918 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚45869490454016 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚45392048046592 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7655151 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0UÐ|ç©2¨- _wandbruntime‚1854  _runtime‚1854V6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2155753770694 _step‚86 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11901720 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚20133264 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚43271678537216 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚43271660534272 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49608322 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚41334512465664 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚45433298109440 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2161870490112 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚29475056 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20133266 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2201783197696 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 _runtime‚1854.654700739 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚45911150483968 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20133264 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 285081600 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚41334529505024 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚49608322 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚698 trainer/global_step‚1099 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11901722 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19562834 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336281088 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚29450184 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2155524796950 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚45433549027328 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚29475056 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2161637575168 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7661114 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104ýŒÈÛüªÍM ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚41334529505024 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚49608322 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚698 trainer/global_step‚1099 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11901722 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19562834 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336281088 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚29450184 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2155524796950 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚45433549027328 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚29475056 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2161637575168 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7661114 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚49583448 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20133266 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2155753770694 _step‚86 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11901720 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚20133264 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚43271678537216 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚43271660534272 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49608322 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚41334512465664 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚45433298109440 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2161870490112 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚29475056 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20133266 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2201783197696 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 _runtime‚1854.654700739 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚45911150483968 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20133264 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 285081600 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2V¤ ~ã2«- _wandbruntime‚1854  _runtime‚1854„»¦¥O¬É 2loss/SRqqt2/assignment_loss‚3.1338979303836823 1loss/SRqqt4/assignment_loss‚3.463044222444296 0loss/FBt3/assignment_loss‚1.0908924440542858 1loss/SRqqt1/detection_loss‚2.8078798154989877 /loss/FBt1/detection_loss‚1.9862617049366236 _step‚87 /loss/FBt1/assignment_loss‚0.777272783270746 1loss/SRqqt4/detection_loss‚2.5785395577549934 .loss/FBt3/detection_loss‚3.199011961619059 _runtime‚1854.658488284 -loss/FRt2/assignment_loss‚5.8026615858078 /loss/FRt3/assignment_loss‚3.595602390666803 .loss/FRt3/detection_loss‚1.796375242786275 /loss/FRt4/detection_loss‚2.5358206266310157 1loss/SRqqt2/detection_loss‚2.9076281897723675 /loss/FRt1/assignment_loss‚5.915702385060928 0loss/FBt2/assignment_loss‚0.5113455874062228 0loss/SRqqt3/detection_loss‚0.903607839345932 epoch‚0 /loss/FRt4/assignment_loss‚4.638667318879104 trainer/global_step‚1099 1loss/FBt4/assignment_loss‚0.32492160209407034 /loss/SRqqt1/assignment_loss‚2.8587739666303 .loss/FRt1/detection_loss‚2.785687572815839 .loss/FRt2/detection_loss‚3.456954574584961 /loss/FBt2/detection_loss‚3.2939970926804976 /loss/FBt4/detection_loss‚0.7545415278248213 2loss/SRqqt3/assignment_loss‚2.4119417684078215 %loss/total_loss‚63.53102969135743 % _timestamp‚1.7885226164383156e+09VRS­Í /loss/SRqqt1/assignment_loss‚2.8587739666303 .loss/FRt1/detection_loss‚2.785687572815839 .loss/FRt2/detection_loss‚3.456954574584961 /loss/FBt2/detection_loss‚3.2939970926804976 /loss/FBt4/detection_loss‚0.7545415278248213 2loss/SRqqt3/assignment_loss‚2.4119417684078215 %loss/total_loss‚63.53102969135743 % _timestamp‚1.7885226164383156e+09 2loss/SRqqt2/assignment_loss‚3.1338979303836823 1loss/SRqqt4/assignment_loss‚3.463044222444296 0loss/FBt3/assignment_loss‚1.0908924440542858 1loss/SRqqt1/detection_loss‚2.8078798154989877 /loss/FBt1/detection_loss‚1.9862617049366236 _step‚87 /loss/FBt1/assignment_loss‚0.777272783270746 1loss/SRqqt4/detection_loss‚2.5785395577549934 .loss/FBt3/detection_loss‚3.199011961619059 _runtime‚1854.658488284 -loss/FRt2/assignment_loss‚5.8026615858078 /loss/FRt3/assignment_loss‚3.595602390666803 .loss/FRt3/detection_loss‚1.796375242786275 /loss/FRt4/detection_loss‚2.5358206266310157 1loss/SRqqt2/detection_loss‚2.9076281897723675 /loss/FRt1/assignment_loss‚5.915702385060928 0loss/FBt2/assignment_loss‚0.5113455874062228 0loss/SRqqt3/detection_loss‚0.903607839345932 epoch‚0 /loss/FRt4/assignment_loss‚4.638667318879104 trainer/global_step‚1099 1loss/FBt4/assignment_loss‚0.32492160209407034W™tû†Ž®‚(:ƒ ýàêÔìþ°÷ gpu.0.gpu‚22.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.14' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.628 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ec’r¯‚(:ç ýàêÔùµµø network.recv‚638744# disk.nvme0c0n1.in‚ 4306.56640625' disk.nvme0c0n1.out‚33978.7197265625% disk.nvme0n1.out‚33978.7197265625 disk.sda1.in‚0 disk.dm-1.out‚ 53.56640625 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0 disk.dm-0.in‚ 4306.56640625" disk.dm-0.out‚33978.7197265625 disk.sda.in‚0 disk.sda3.in‚0# memory_percent‚96.5902328491211% proc.memory.availableMB‚ 558.65625# proc.memory.rssMB‚ 2814.57421875 network.sent‚ 1.754219e+06) disk./.usagePercent‚78.94292896745321% disk./.usageGB‚1176.3066902160645 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.2578125 disk.sda.out‚ 53.56640625 disk.sda3.out‚ 53.56640625! disk.nvme0n1.in‚ 4306.56640625* proc.memory.percent‚17.178797721862793 cpu‚25.18000350279262 proc.cpu.threads‚17Ä0êÿ‘°‚(:† ŒáêÔáÔðö gpu.0.gpu‚20.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚102.662' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.5324 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚05Çk±‚(:à ŒáêÔœÙÓù disk.sda1.in‚0# proc.memory.rssMB‚ 2814.57421875 cpu‚25.17947684052706 disk.sda3.in‚0! disk.nvme0n1.in‚ 4306.56640625 disk.sda1.out‚0 disk.dm-1.out‚54.1875 disk.sda2.in‚0 disk.dm-0.in‚ 4306.56640625 disk.sda.in‚0( proc.memory.availableMB‚ 558.85546875 network.sent‚ 1.806379e+06 network.recv‚644971# disk.nvme0c0n1.in‚ 4306.56640625 disk.drbd1260.out‚ 0.26171875 disk.sda2.out‚0" disk.dm-0.out‚34361.6142578125$ memory_percent‚96.58901691436768 proc.cpu.threads‚17 disk.sda3.out‚54.1875' disk.nvme0c0n1.out‚34361.6142578125% disk.nvme0n1.out‚34361.6142578125 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda.out‚54.1875* proc.memory.percent‚17.178797721862793) disk./.usagePercent‚78.94135400910255% disk./.usageGB‚1176.2832221984863/ŽŸå‘²‚(:† ›áêÔ¨‚°÷ gpu.0.gpu‚25.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚106.541' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.3082 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0…ÏnFw³‚(:ì ›áêÔÙ€˜û disk.sda2.out‚0 disk.sda1.in‚0 disk.drbd1260.out‚ 0.26171875" disk.dm-0.out‚34660.8564453125# disk.nvme0c0n1.in‚ 4306.56640625# proc.memory.rssMB‚ 2814.57421875* proc.memory.percent‚17.178797721862793 cpu‚25.179208075320155 network.sent‚ 1.809409e+06! disk.nvme0n1.in‚ 4306.56640625 disk.sda2.in‚0 disk.dm-0.in‚ 4306.56640625 disk.sda.in‚0 disk.sda.out‚ 54.45703125' disk.nvme0c0n1.out‚34660.8564453125# memory_percent‚96.5907335281372 network.recv‚648368) disk./.usagePercent‚78.94135528914023% disk./.usageGB‚1176.2832412719727 disk.sda1.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 54.45703125 disk.sda3.out‚ 54.45703125( proc.memory.availableMB‚ 558.57421875% disk.nvme0n1.out‚34660.8564453125 disk.drbd1260.in‚0 disk.sda3.in‚0 proc.cpu.threads‚17uÊÌ•f´‚(:Û ªáêÔº«¸÷ cpu‚25.178828524810434 network.sent‚ 1.814739e+06 network.recv‚651926% disk./.usageGB‚1176.3066902160645 disk.sda2.in‚0 disk.dm-0.out‚34887.52734375 disk.sda.in‚0# disk.nvme0c0n1.in‚ 4306.56640625 disk.dm-1.in‚0) disk./.usagePercent‚78.94292896745321 disk.sda2.out‚0 disk.dm-1.out‚ 54.828125 disk.drbd1260.in‚0 disk.drbd1260.out‚0.265625$ memory_percent‚96.59080505371094 disk.sda.out‚ 54.828125! disk.nvme0n1.in‚ 4306.56640625# disk.nvme0n1.out‚34887.52734375$ proc.memory.availableMB‚558.5625* proc.memory.percent‚17.178797721862793 proc.cpu.threads‚17 disk.dm-0.in‚ 4306.56640625 disk.sda3.in‚0 disk.sda3.out‚ 54.828125% disk.nvme0c0n1.out‚34887.52734375 disk.sda1.in‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 2814.57421875¾øAµ‚(:… ªáêÔŵø gpu.0.gpu‚44.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚111.635' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.327 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Fgꑶ‚(:† ¹áêÔܱ÷ gpu.0.gpu‚44.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚113.954' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.7908 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0+èÕhe·‚(:Ú ¹áêÔîÒòú disk.sda.out‚ 55.1484375& proc.memory.availableMB‚ 558.796875 disk.sda2.in‚0 disk.sda.in‚0% disk.nvme0c0n1.out‚35352.89453125# proc.memory.rssMB‚ 2814.57421875* proc.memory.percent‚17.178797721862793 proc.cpu.threads‚17 disk.sda2.out‚0 disk.dm-0.out‚35352.89453125 disk.sda3.in‚0 disk.sda3.out‚ 55.1484375! disk.nvme0c0n1.in‚ 4450.984375$ memory_percent‚96.58937454223633 cpu‚25.178771493375955 network.recv‚656344) disk./.usagePercent‚78.94292896745321# disk.nvme0n1.out‚35352.89453125 disk.sda1.out‚0 disk.dm-1.in‚0 disk.drbd1260.out‚0.265625 disk.dm-0.in‚ 4450.984375 network.sent‚ 1.820481e+06% disk./.usageGB‚1176.3066902160645 disk.nvme0n1.in‚ 4450.984375 disk.sda1.in‚0 disk.dm-1.out‚ 55.1484375 disk.drbd1260.in‚0ßßý¸‚(:„ ÈáêÔÒâˆ÷ gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚29.0 gpu.0.powerWatts‚92.744' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.5488 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0漬a¹‚(:Ö ÈáêÔ€Ñ÷ cpu‚25.17838777111996 disk.sda3.out‚55.78125! disk.nvme0c0n1.in‚ 4450.984375 disk.nvme0n1.in‚ 4450.984375 disk.drbd1260.in‚0$ memory_percent‚96.59185409545898* proc.memory.percent‚17.178797721862793 proc.cpu.threads‚17 network.recv‚661061 disk.sda.in‚0& disk.nvme0c0n1.out‚35579.591796875$ disk.nvme0n1.out‚35579.591796875 disk.sda2.in‚0 disk.dm-0.in‚ 4450.984375! disk.dm-0.out‚35579.591796875) disk./.usagePercent‚78.94291872715185 disk.sda3.in‚0 disk.sda1.out‚0 disk.dm-1.out‚55.78125 disk.sda2.out‚0& proc.memory.availableMB‚ 558.390625# proc.memory.rssMB‚ 2814.57421875 network.sent‚ 1.826697e+06% disk./.usageGB‚1176.3065376281738 disk.sda.out‚55.78125 disk.sda1.in‚0 disk.dm-1.in‚0 disk.drbd1260.out‚0.265625¢k2-2º- _wandbruntime‚1940  _runtime‚1940äH 4Å»¿ $ _timestamp‚1.788522702106997e+09 _runtime‚1940.326696548 _step‚88 & lr-AdamW/pg1‚0.0008732790623895033 & lr-AdamW/pg2‚0.0008732790623895033 trainer/global_step‚1149CL%Zɼà _runtime‚1940.326696548 _step‚88 & lr-AdamW/pg1‚0.0008732790623895033 & lr-AdamW/pg2‚0.0008732790623895033 trainer/global_step‚1149 $ _timestamp‚1.788522702106997e+09XYªX72½- _wandbruntime‚1940  _runtime‚1940ŒLÏ@Å'¾¿O ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚45196676346880 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚21028739 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚43173517018800 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚1940.327852717 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚21028741 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚51815031 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚50253405171200 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚45425250340808 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚47454565602816 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚47454565602816 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2258122157568 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20437440 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 308438528 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚43173499979440 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚30786290 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚30761444 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚45196694349824 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚45196694349824 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8005719 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21028739 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚50253720497664 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2299740154880 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12431020 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚47953688956416 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2251504348472 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8006420 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 291386368 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2258122157568 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2257889255936 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12431018 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚47454816507392 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚701 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚45425004327912 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2300031541248 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608¤ÉÈ$¿ÃO JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 trainer/global_step‚1149 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚703 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚45196676346880 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 315326464 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚47454565602816 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2258122157568 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20437440 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 308438528 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚43173499979440 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚30786290 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚30761444 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚45196694349824 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚45196694349824 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8005719 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21028739 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚50253720497664 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2299740154880 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12431020 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚47953688956416 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2251504348472 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8006420 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 291386368 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2258122157568 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2257889255936 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12431018 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚47454816507392 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚701 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚45425004327912 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2300031541248 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚45196676346880 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚21028739 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚43173517018800 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚1940.327852717 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚21028741 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚51815031 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚50253405171200 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚45425250340808 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚47454565602816 IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2259853281792 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚51860024 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚51835150 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚47995348986368 % _timestamp‚1.7885227031348937e+09 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚691 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 313216000 _step‚90 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 289275904 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚45235980386816 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336281088 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚30813008 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚47496066583552 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚43211030336864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 _runtime‚1941.356187206 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21047016 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2260086196736 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚47496066583552 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12441822 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8012752 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20454572 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2259853281792 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚51860024 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0š¢)Ó&ÂÍM GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚47995372926464 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚45235980386816 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336281088 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚30813008 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚47496066583552 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚43211030336864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 _runtime‚1941.356187206 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21047016 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246013104 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2260086196736 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚47496066583552 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12441822 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8012752 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20454572 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2259853281792 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚51860024 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚45235962383872 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚45235962383872 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚45464493174322 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚43211047376224 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2259853281792 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚51860024 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚51835150 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚47995348986368 % _timestamp‚1.7885227031348937e+09 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚691 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 313216000 _step‚90 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 289275904 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 Od2WÅÑ % _timestamp‚1.7885227031373785e+09 0loss/FRt3/assignment_loss‚2.3499713776967464 0loss/FBt2/assignment_loss‚0.5114958301031342 1loss/SRqqt2/detection_loss‚2.6691827859197343 2loss/SRqqt4/assignment_loss‚2.3578848558313705 /loss/FBt4/detection_loss‚0.9417554974618058 /loss/FRt2/assignment_loss‚6.134510056725864 1loss/SRqqt2/assignment_loss‚2.775051989725658 /loss/FBt2/detection_loss‚2.9630939265092215 epoch‚0 1loss/SRqqt4/detection_loss‚2.8751340718830334 trainer/global_step‚1149 /loss/FRt1/assignment_loss‚6.274736841519673 0loss/FRt4/assignment_loss‚3.1526696073512235 -loss/FRt3/detection_loss‚1.56558324664067 .loss/FBt1/detection_loss‚2.711003489792347 %loss/total_loss‚59.70545148343441 _step‚91 1loss/SRqqt3/assignment_loss‚2.139575636828387 .loss/FRt2/detection_loss‚3.045775791694378 /loss/FRt4/detection_loss‚1.9751721046244104 1loss/SRqqt1/detection_loss‚2.7965154868105184 1loss/SRqqt3/detection_loss‚0.8927245264252027 .loss/FBt3/detection_loss‚5.381647109985352 .loss/FRt1/detection_loss‚2.352721028857761 0loss/FBt1/assignment_loss‚0.6496756686756271 0loss/FBt3/assignment_loss‚0.1696896106004715 _runtime‚ 1941.35756344 1loss/SRqqt1/assignment_loss‚2.776077536137208 1loss/FBt4/assignment_loss‚0.24380340563460778[¦™+Æ‚(:‚ ×áêÔ¸Ïàø gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.14' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.628 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0®[ýâlÇ‚(:á ×áêԾػú) disk./.usagePercent‚78.94137576974298 disk.sda2.out‚0 disk.sda.out‚ 56.203125$ memory_percent‚96.59066200256348# proc.memory.rssMB‚ 2814.57421875* proc.memory.percent‚17.178797721862793 cpu‚25.177851671687947 proc.cpu.threads‚17 network.recv‚664810! disk.nvme0c0n1.in‚ 4450.984375( disk.nvme0c0n1.out‚35896.83349609375 disk.nvme0n1.in‚ 4450.984375& disk.nvme0n1.out‚35896.83349609375 disk.sda1.in‚0 disk.drbd1260.in‚0 disk.sda.in‚0 network.sent‚ 1.83171e+06$ disk./.usageGB‚1176.283546447754 disk.dm-1.in‚0 disk.dm-1.out‚ 56.203125 disk.drbd1260.out‚ 0.26953125# disk.dm-0.out‚35896.83349609375 disk.sda3.in‚0 disk.sda3.out‚ 56.203125 disk.sda1.out‚0 disk.sda2.in‚0 disk.dm-0.in‚ 4450.984375' proc.memory.availableMB‚ 558.5859375ÓaÔaÈ‚(:„ æáêÔ¨êýö gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚98.552' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.7104 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0˜°ÊuÉ‚(:ê æáêÔì€ßø disk.sda.out‚ 56.51171875) disk./.usagePercent‚78.93979953135467 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 56.51171875$ memory_percent‚96.59068584442139( proc.memory.availableMB‚ 558.58203125% disk./.usageGB‚1176.2600593566895! disk.nvme0c0n1.in‚ 4450.984375 disk.nvme0n1.in‚ 4450.984375 disk.sda2.out‚0# disk.dm-0.out‚36272.84521484375# proc.memory.rssMB‚ 2814.57421875* proc.memory.percent‚17.178797721862793 network.sent‚ 1.880613e+06& disk.nvme0n1.out‚36272.84521484375 disk.sda.in‚0 cpu‚25.177564916642176 proc.cpu.threads‚17 disk.drbd1260.out‚ 0.26953125 network.recv‚669807( disk.nvme0c0n1.out‚36272.84521484375 disk.dm-1.in‚0 disk.dm-1.out‚ 56.51171875 disk.drbd1260.in‚0 disk.dm-0.in‚ 4450.9843752´œÊ‚(:‘ õáêÔð¾Àö gpu.0.gpu‚19.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚103.766' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚20.753200000000003 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0±©*lË‚(:á õáêÔ­§òø% disk./.usageGB‚1176.3069953918457 disk.sda1.in‚0 disk.drbd1260.in‚0 disk.sda3.in‚0" disk.nvme0c0n1.in‚ 4589.9140625# proc.memory.rssMB‚ 2814.57421875 proc.cpu.threads‚17 disk.sda.out‚56.90625( disk.nvme0c0n1.out‚36531.93505859375 disk.nvme0n1.in‚ 4589.9140625# memory_percent‚96.5907335281372* proc.memory.percent‚17.178797721862793 network.sent‚ 1.886666e+06 disk.sda1.out‚0 disk.dm-1.in‚0# disk.dm-0.out‚36531.93505859375 disk.sda.in‚0& disk.nvme0n1.out‚36531.93505859375( proc.memory.availableMB‚ 558.57421875 cpu‚25.17723408801053 network.recv‚674077) disk./.usagePercent‚78.94294944805594 disk.dm-1.out‚56.90625 disk.drbd1260.out‚ 0.2734375 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 4589.9140625 disk.sda3.out‚56.90625ãn¾šÌ‚(: „âêÔ›â±ö gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.059' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚18.611800000000002 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ù¹.tÍ‚(:é „âêÔ Öúø% disk.nvme0n1.out‚36786.7353515625 disk.sda1.in‚0 disk.sda1.out‚0" disk.dm-0.out‚36786.7353515625 disk.sda.in‚0( proc.memory.availableMB‚ 558.33203125# proc.memory.rssMB‚ 2814.57421875% disk./.usageGB‚1176.3069953918457 disk.sda3.in‚0 disk.sda3.out‚ 57.48828125 disk.dm-1.in‚0 disk.dm-1.out‚ 57.48828125 disk.drbd1260.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 4589.9140625) disk./.usagePercent‚78.94294944805594" disk.nvme0c0n1.in‚ 4589.9140625' disk.nvme0c0n1.out‚36786.7353515625 disk.drbd1260.out‚ 0.2734375 disk.sda.out‚ 57.48828125* proc.memory.percent‚17.178797721862793 proc.cpu.threads‚17 network.sent‚ 1.892783e+06 disk.nvme0n1.in‚ 4589.9140625 disk.sda2.in‚0$ memory_percent‚96.59221172332764 cpu‚25.176788641732134 network.recv‚678943 ´Ör‘΂(:† “âêÔ߶’÷ gpu.0.gpu‚35.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚108.058' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.6116 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0áÏÏ‚(:â “âêÔæŸù) disk./.usagePercent‚78.94295072809362 disk.drbd1260.in‚0 disk.sda2.out‚0$ memory_percent‚96.5922594ÌñÀŽÝ0704346# proc.memory.rssMB‚ 2814.57421875* proc.memory.percent‚17.178797721862793 cpu‚25.17658615439368$ disk.nvme0n1.out‚37103.861328125 disk.sda1.out‚0 disk.dm-1.out‚ 57.8359375 disk.drbd1260.out‚ 0.27734375 disk.dm-0.in‚ 4589.9140625 disk.sda3.in‚0$ disk./.usageGB‚1176.307014465332" disk.nvme0c0n1.in‚ 4589.9140625 disk.nvme0n1.in‚ 4589.9140625 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda.out‚ 57.8359375 disk.sda3.out‚ 57.8359375 proc.cpu.threads‚17 network.sent‚ 1.899734e+06 network.recv‚682112& disk.nvme0c0n1.out‚37103.861328125 disk.sda1.in‚0 disk.sda2.in‚0! disk.dm-0.out‚37103.861328125( proc.memory.availableMB‚ 558.32421875c­*hnЂ(:ã ¢âêÔ£öÂù disk.dm-1.in‚0 disk.dm-1.out‚ 58.06640625 disk.drbd1260.in‚0# memory_percent‚96.5909481048584) disk./.usagePercent‚78.94216094485049" disk.nvme0c0n1.in‚ 4589.9140625& disk.nvme0c0n1.out‚37478.673828125 disk.drbd1260.out‚ 0.27734375 disk.sda2.out‚0 disk.dm-0.in‚ 4589.9140625! disk.dm-0.out‚37478.673828125 disk.sda.in‚0 network.sent‚ 1.90292e+06 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 58.06640625' proc.memory.availableMB‚ 558.5390625* proc.memory.percent‚17.178797721862793 proc.cpu.threads‚17 network.recv‚685496$ disk.nvme0n1.out‚37478.673828125 disk.sda2.in‚0 disk.sda.out‚ 58.06640625# proc.memory.rssMB‚ 2814.57421875 cpu‚25.17614639554923% disk./.usageGB‚1176.2952461242676 disk.nvme0n1.in‚ 4589.9140625ˆgÌ ‘Ñ‚(:† ¢âêÔ¢ÙÔù gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚133.326' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.6652 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0lÐ:Ÿ2Ò- _wandbruntime‚2028  _runtime‚2028üMd\ÆÓÀ & lr-AdamW/pg2‚0.0009112807622323886 trainer/global_step‚1199 % _timestamp‚1.7885227903313816e+09 _runtime‚2028.551089075 _step‚92 & lr-AdamW/pg1‚0.0009112807622323886o}ÏÊÔÄ & lr-AdamW/pg1‚0.0009112807622323886 & lr-AdamW/pg2‚0.0009112807622323886 trainer/global_step‚1199 % _timestamp‚1.7885227903313816e+09 _runtime‚2028.551089075 _step‚92\—P…2Õ- _wandbruntime‚2028  _runtime‚20284ˆL±Å'Ö¿O E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8356120 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2356337864192 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚738 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚49517334063616 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚47160978196480 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚21942491 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚49517083159040 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚52437711969792 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚50037887458816 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 trainer/global_step‚1199 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚50037863518720 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚45050034890000 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 287192064 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚45050017850640 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚47160978196480 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2356337864192 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚21942491 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 308438528 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚47399706252516 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21942489 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12971118 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2400135643136 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12971120 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚32124242 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚47160996199424 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21327240 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚32099396 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚93 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚49517083159040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21326502 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚32124242 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚32099396 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2349442388980 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2356104962560 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚21942489 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚47160996199424 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8355384 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336281088 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2399848451072 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 % _timestamp‚1.7885227903330166e+09 _runtime‚2028.552753662 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚52438023101952 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚736 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚49517083159040 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚52437711969792 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚50037887458816 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 trainer/global_step‚1199 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30751 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966133760 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚50037863518720 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚45050034890000 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 564258816 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 287192064 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚45050017850640 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 232901632 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚47160978196480 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2356337864192 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 232901632 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚21942491 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24846 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246012896 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24848 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 308438528 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚47399706252516 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228973536 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21942489 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12971118 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2400135643136 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12971120 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚32124242 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚47160996199424 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 250904576 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21327240 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741051909 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 250904576 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚32099396 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚93 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚49517083159040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30751 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21326502 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24848 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚32124242 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚32099396 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2349442388980 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2356104962560 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30508 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚21942489 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚47160996199424 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8355384 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 336281088 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2399848451072 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 % _timestamp‚1.7885227903330166e+09 _runtime‚2028.552753662 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚52438023101952 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560281229 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24846 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚736 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30508 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2356337864192 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 564258816 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚738 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚49517334063616 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚47160978196480 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚21942491]J Ýî2Ø- _wandbruntime‚2030  _runtime‚2030d·íÎ#&ÙÉM ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚47200282236416 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 564258816 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21960766 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30508 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21344182 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚54086852 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚730 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚50079571428864 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21960766 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336281088 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚47438949086030 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 _runtime‚2030.662596285 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚54111726 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 299761664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 308438528 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚45087548208064 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 % _timestamp‚1.7885227924370627e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚32150960 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚49558584139776 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚32126088 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚52481705842176 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚32126088 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚54111726 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚49558333221888 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚47200282236416 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚1199 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚49558584139776 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚45087565247424 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2351400877966 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚32150960 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚50079547488768 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2401834651648 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2358068988416 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12981922 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8362262 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8362992 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 323701760 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21344914 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12981920 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚47200264233472 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚54086852 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2358068988416 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2402134413312 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21344914 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12981920 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚47200264233472 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚54086852 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 250917888 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741051909 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2358068988416 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2402134413312 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21344182 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚54086852 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24872 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚730 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚50079571428864 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 250917888 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30751 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21960766 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966133760 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 336281088 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚47438949086030 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 _runtime‚2030.662596285 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚54111726 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 299761664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 308438528 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚45087548208064 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 % _timestamp‚1.7885227924370627e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚32150960 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚49558584139776 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚32126088 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 564258816 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚52481705842176 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228973744 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚32126088 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚54111726 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚49558333221888 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚47200282236416 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560281229 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚1199 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚49558584139776 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚45087565247424 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2351400877966 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966133760 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚32150960 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚50079547488768 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2401834651648 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 232914944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2358068988416 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12981922 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8362262 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8362992 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 323701760 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24874 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24872 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 232914944 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136^>~D¤2Û- _wandbruntime‚2030  _runtime‚2030ŽŸPÜÊ /loss/FRt2/detection_loss‚3.9419933908125935 0loss/SRqqt4/detection_loss‚2.654589452913829 -loss/FBt1/detection_loss‚3.10423743724823 /loss/FBt4/detection_loss‚0.5422520829035955 2loss/SRqqt2/assignment_loss‚2.7255272504055137 /loss/FRt3/detection_loss‚1.6759795848582242 %loss/total_loss‚60.02968606956147 _runtime‚2030.666379515 /loss/FRt3/assignment_loss‚3.142103171831853 /loss/FRt4/assignment_loss‚3.410329871582535 0loss/FBt3/assignment_loss‚0.8127228418986002 1loss/SRqqt3/detection_loss‚0.8153211121149615 epoch‚0 2loss/SRqqt4/assignment_loss‚2.3814559589539255 1loss/SRqqt2/detection_loss‚2.3616893715930707 % _timestamp‚1.7885227924456398e+09 /loss/SRqqt1/detection_loss‚2.76196410168301 /loss/FRt1/assignment_loss‚5.568833651366057 0loss/FBt2/assignment_loss‚0.4968740192675341 0loss/FBt4/assignment_loss‚0.2925818242935587 /loss/FRt1/detection_loss‚3.1024710646382085 trainer/global_step‚1199 _step‚95 /loss/FRt2/assignment_loss‚5.805460200590246 .loss/SRqqt3/assignment_loss‚2.196684869171 /loss/FRt4/detection_loss‚1.9342734954267178 .loss/FBt2/detection_loss‚2.642735940217972 /loss/FBt3/detection_loss‚3.8814229170481362 2loss/SRqqt1/assignment_loss‚2.9939034818248316 0loss/FBt1/assignment_loss‚0.7842789769172669>åÖ‰TÝÎ /loss/FRt2/assignment_loss‚5.805460200590246 .loss/SRqqt3/assignment_loss‚2.196684869171 /loss/FRt4/detection_loss‚1.9342734954267178 .loss/FBt2/detection_loss‚2.642735940217972 /loss/FBt3/detection_loss‚3.8814229170481362 2loss/SRqqt1/assignment_loss‚2.9939034818248316 0loss/FBt1/assignment_loss‚0.7842789769172669 /loss/FRt2/detection_loss‚3.9419933908125935 0loss/SRqqt4/detection_loss‚2.654589452913829 -loss/FBt1/detection_loss‚3.10423743724823 /loss/FBt4/detection_loss‚0.5422520829035955 2loss/SRqqt2/assignment_loss‚2.7255272504055137 /loss/FRt3/detection_loss‚1.6759795848582242 %loss/total_loss‚60.02968606956147 _runtime‚2030.666379515 /loss/FRt3/assignment_loss‚3.142103171831853 /loss/FRt4/assignment_loss‚3.410329871582535 0loss/FBt3/assignment_loss‚0.8127228418986002 1loss/SRqqt3/detection_loss‚0.8153211121149615 epoch‚0 2loss/SRqqt4/assignment_loss‚2.3814559589539255 1loss/SRqqt2/detection_loss‚2.3616893715930707 % _timestamp‚1.7885227924456398e+09 /loss/SRqqt1/detection_loss‚2.76196410168301 /loss/FRt1/assignment_loss‚5.568833651366057 0loss/FBt2/assignment_loss‚0.4968740192675341 0loss/FBt4/assignment_loss‚0.2925818242935587 /loss/FRt1/detection_loss‚3.1024710646382085 trainer/global_step‚1199 _step‚95_§Û~cÞ‚(:„ ±âêÔ© ƒø gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚92.852' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.5704 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0éb&Sß‚(:È ±âêÔ¿¼íø disk.sda3.out‚ 58.37890625# disk.nvme0c0n1.out‚ 37795.921875! disk.nvme0n1.out‚ 37795.921875 disk.dm-1.out‚ 58.37890625 disk.drbd1260.in‚0# proc.memory.rssMB‚ 2814.57421875 network.sent‚ 1.90789e+06 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0 disk.sda2.in‚0 disk.dm-0.in‚4736.5 disk.sda.in‚0* proc.memory.percent‚17.178797721862793 disk.nvme0n1.in‚4736.5 disk.sda.out‚ 58.37890625( proc.memory.availableMB‚ 558.25390625 proc.cpu.threads‚17 network.recv‚689128% disk./.usageGB‚1176.3069648742676 disk.nvme0c0n1.in‚4736.5 disk.drbd1260.out‚0.28125 disk.sda2.out‚0 disk.dm-0.out‚ 37795.921875$ memory_percent‚96.59268856048584 cpu‚25.175975352735996) disk./.usagePercent‚78.94294739999567 disk.sda3.in‚0òß9‘à‚(:† ÀâêÔ³Öò÷ gpu.0.gpu‚21.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚102.421' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.4842 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0O›ªMá‚(: ÀâêÔÊàøø disk.dm-0.out‚ 38022.34375 disk.sda.out‚ 58.88671875 disk.sda3.in‚0 disk.nvme0n1.in‚4736.5 disk.nvme0n1.out‚ 38022.34375 disk.dm-1.in‚0 network.sent‚ 1.957734e+06 disk.sda.in‚0" proc.memory.rssMB‚ 2903.9765625 proc.cpu.threads‚17 disk.sda3.out‚ 58.88671875) disk./.usagePercent‚78.94295098410116% disk./.usageGB‚1176.3070182800293 disk.sda2.out‚0 disk.dm-0.in‚4736.5 disk.dm-1.out‚ 58.88671875) proc.memory.percent‚17.72446632385254 cpu‚25.1837412040981 network.recv‚694585 disk.nvme0c0n1.in‚4736.5 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚0.28125$ memory_percent‚97.13947772979736( proc.memory.availableMB‚ 468.66796875" disk.nvme0c0n1.out‚ 38022.34375 disk.sda2.in‚0Ÿ$¹–â‚(:… ÏâêÔõ—«ý gpu.0.gpu‚19.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚92.922' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.5844 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 Mk¢]ã‚(:Ò ÏâêÔ‡íÆŠ" disk.dm-0.out‚38367.7333984375 disk.sda3.in‚0) proc.memory.percent‚17.72446632385254 proc.cpu.threads‚17 network.sent‚ 1.964536e+06 disk.nvme0c0n1.in‚4736.5 disk.drbd1260.in‚0 disk.dm-0.in‚4736.5 disk.sda.in‚0 disk.sda3.out‚ 59.37109375 cpu‚25.210881084792657 network.recv‚697812% disk./.usageGB‚1176.3070182800293 disk.drbd1260.out‚ 0.28515625 disk.sda2.in‚0 disk.sda.out‚ 59.37109375% proc.memory.availableMB‚ 469.15625) disk./.usagePercent‚78.94295098410116 disk.nvme0n1.in‚4736.5% disk.nvme0n1.out‚38367.7333984375 disk.sda1.in‚0 disk.dm-1.in‚0 disk.sda2.out‚0# memory_percent‚97.1364974975586" proc.memory.rssMB‚ 2903.9765625' disk.nvme0c0n1.out‚38367.7333984375 disk.sda1.out‚0 disk.dm-1.out‚ 59.37109375\ðÝ‘ä‚(:† ÞâêÔÄÎßõ gpu.0.gpu‚45.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚113.934' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.7868 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/Ûqaå‚(:Ö ÞâêÔçµë÷ disk.nvme0n1.in‚4736.5 disk.sda1.in‚0 disk.dm-1.in‚0) proc.memory.percent‚17.67094135284424% disk./.usageGB‚1176.1496391296387 disk.sda2.in‚0 disk.dm-0.in‚4736.5" disk.dm-0.out‚38464.1708984375 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.28515625$ memory_percent‚97.08883762359619 disk.sda.out‚ 59.60546875 disk.sda3.in‚0 disk.dm-1.out‚ 59.60546875( proc.memory.availableMB‚ 476.96484375# proc.memory.rssMB‚ 2895.20703125 cpu‚25.23712922679782 network.sent‚ 1.967889e+06) disk./.usagePercent‚78.93238913726853 disk.sda2.out‚0 disk.sda.in‚0 disk.sda3.out‚ 59.60546875 disk.nvme0c0n1.in‚4736.5% disk.nvme0n1.out‚38464.1708984375 proc.cpu.threads‚17 network.recv‚701130' disk.nvme0c0n1.out‚38464.1708984375ÄS›æ‚(: íâêÔŽÅ„÷ gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚114.067' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.813399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹_$êsç‚(:è íâêÔÄìêú disk.dm-0.in‚ 4779.5859375 disk.sda.out‚ 59.94921875 disk.sda3.in‚0' proc.memory.availableMB‚ 482.2734375 disk.drbd1260.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0# disk.dm-0.out‚39171.64404296875 disk.sda.in‚0$ memory_percent‚97.05643653869629 cpu‚25.26980229303686 proc.cpu.threads‚17 network.sent‚ 1.973759e+06 disk.sda3.out‚ 59.94921875" proc.memory.rssMB‚ 2890.1953125* proc.memory.percent‚17.640352249145508 disk.drbd1260.out‚ 0.2890625 network.recv‚705526) disk./.usagePercent‚78.90079831955835" disk.nvme0c0n1.in‚ 4779.5859375( disk.nvme0c0n1.out‚39171.64404296875& disk.nvme0n1.out‚39171.64404296875 disk.dm-1.out‚ 59.94921875 disk.sda2.in‚0$ disk./.usageGB‚1175.678913116455 disk.nvme0n1.in‚ 4779.5859375 disk.sda1.in‚0 disk.sda2.out‚0¹ˆ8‘è‚(:† üâêÔ¢Œ¿û gpu.0.gpu‚21.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚112.874' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.5748 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ƒBsrvé‚(:ë üâêÔÀ¡È€ disk.dm-1.in‚0 disk.dm-0.in‚ 4852.76171875 disk.sda.out‚ 60.53515625$ memory_percent‚97.05936908721924* proc.memory.percent‚17.643284797668457 cpu‚25.306456794712958 network.sent‚ 1.98185e+06% disk./.usageGB‚1175.6555480957031 disk.sda1.in‚0 disk.dm-1.out‚ 60.53515625 disk.drbd1260.in‚0" disk.dm-0.out‚39340.3896484375 disk.sda3.in‚0 disk.sda3.out‚ 60.53515625 disk.sda1.out‚0 disk.drbd1260.out‚ 0.2890625 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda.in‚0' disk.nvme0c0n1.out‚39340.3896484375! disk.nvme0n1.in‚ 4852.76171875% disk.nvme0n1.out‚39340.3896484375 network.recv‚710087# disk.nvme0c0n1.in‚ 4852.76171875( proc.memory.availableMB‚ 481.79296875# proc.memory.rssMB‚ 2890.67578125 proc.cpu.threads‚17) disk./.usagePercent‚78.89923027341113¢¥Éyêbt QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak"trainer/global_step2:JPÜôËÃyëbt QDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated"trainer/global_step2:JPŸö~Nlìbg DDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak"trainer/global_step2:JPèwÍdíb_ DeviceStatsMonitor.on_validation_batch_start/segment.all.freed"trainer/global_step2:JPuÿbp MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed"trainer/global_step2:JP?ÀÓ6w€br ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current"trainer/global_step2:JP‹LL\eb` =DeviceStatsMonitor.on_validation_batch_start/active.all.freed"trainer/global_step2:JPß#l‚bg DDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed"trainer/global_step2:JPK½w7qƒbl IDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current"trainer/global_step2:JPiµ8t„bo LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak"trainer/global_step2:JPqaœ¦h…bc @DeviceStatsMonitor.on_validation_batch_start/segment.all.current"trainer/global_step2:JP*Vto†bj GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated"trainer/global_step2:JPy ‚ k‡bf CDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed"trainer/global_step2:JPFÕ:=sˆbn KDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak"trainer/global_step2:JP¨-‘Ô~‰by VDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated"trainer/global_step2:JPŒF©G]ŠbX 5DeviceStatsMonitor.on_validation_batch_start/num_ooms"trainer/global_step2:JP$1·n‹bi FDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak"trainer/global_step2:JPGÀÌlŒbg DDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed"trainer/global_step2:JP»ØO°gbb ?DeviceStatsMonitor.on_validation_batch_start/active.all.current"trainer/global_step2:JPi(ÀUtŽbo LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak"trainer/global_step2:JPF´íybt QDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated"trainer/global_step2:JP»3“Ùmbh EDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak"trainer/global_step2:JP¹Â®¼s‘bn KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak"trainer/global_step2:JP€äÚ¶v’bq NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current"trainer/global_step2:JP UXBk“bf CDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak"trainer/global_step2:JP!¦y”bt QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak"trainer/global_step2:JP"?#±p•bk HDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated"trainer/global_step2:JP§Ø6ït–bo LDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current"trainer/global_step2:JPǹ©o—bj GDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current"trainer/global_step2:JP½0Üt˜bo LDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed"trainer/global_step2:JPDƒJ–r™bm JDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak"trainer/global_step2:JPÙ­uÀ|šbw TDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current"trainer/global_step2:JPAh8áw›br ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current"trainer/global_step2:JPÇñÜqœbl IDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated"trainer/global_step2:JP¿_ì>obj GDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak"trainer/global_step2:JPu ±3hžbc @DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak"trainer/global_step2:JPŒ€ÀtŸbo LDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed"trainer/global_step2:JPô[£~ by VDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated"trainer/global_step2:JP4u¡bp MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed"trainer/global_step2:JPg8­zv¢bq NDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current"trainer/global_step2:JP‚èÓq£bl IDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated"trainer/global_step2:JPp¹Bl¤bg DDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak"trainer/global_step2:JPÍá¤t¥bo LDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated"trainer/global_step2:JP˜!ëq¦bl IDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated"trainer/global_step2:JP5óa'p§bk HDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed"trainer/global_step2:JPÅÊ­¶z¨bu RDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed"trainer/global_step2:JPÍ–œd©b_ DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries"trainer/global_step2:JP£Mk²bf CDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak"trainer/global_step2:JPU׫t³bo LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak"trainer/global_step2:JPØ!€t´bo LDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed"trainer/global_step2:JP6Gó—mµbh EDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed"trainer/global_step2:JP£M¼wg¶bb ?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections"trainer/global_step2:JPHKÎp·bk HDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated"trainer/global_step2:JPLÔp¸bk HDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current"trainer/global_step2:JPc«fSn¹bi FDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed"trainer/global_step2:JPØLyGxºbs PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated"trainer/global_step2:JP¡»ÙÀu»bp MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current"trainer/global_step2:JPýcs¼bn KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed"trainer/global_step2:JPuÔ(o½bj GDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed"trainer/global_step2:JP-[or¾bm JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed"trainer/global_step2:JP½ózr¿bm JDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current"trainer/global_step2:JPÊþNceÀb` =DeviceStatsMonitor.on_validation_batch_start/num_device_alloc"trainer/global_step2:JP‚ú¤mÁbh EDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak"trainer/global_step2:JPûÂHjÂbe BDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated"trainer/global_step2:JP^oaHwÃbr ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current"trainer/global_step2:JP»ú#ítÄbo LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak"trainer/global_step2:JP+kÓwÅbr ODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated"trainer/global_step2:JP9x‘¶iÆbd ADeviceStatsMonitor.on_validation_batch_start/active.all.allocated"trainer/global_step2:JP÷ldrÇbm JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated"trainer/global_step2:JP² wÈbr ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current"trainer/global_step2:JP¹P¡&pÉbk HDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed"trainer/global_step2:JPÛèkQuÊbp MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed"trainer/global_step2:JPxúÀ=xËbs PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated"trainer/global_step2:JPź%oÌbj GDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current"trainer/global_step2:JP33÷vÍbq NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated"trainer/global_step2:JPy:urÎbm JDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current"trainer/global_step2:JPæ 0¾CÏbg DDeviceStatsMonitor.on_validation_batch_start/inactive_split.ÿz)all.peak"trainer/global_step2:JP±~ÌÃxÐbs PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated"trainer/global_step2:JPÔ°…sÑbn KDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated"trainer/global_step2:JPçlÒbg DDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak"trainer/global_step2:JP&–JmÓbh EDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed"trainer/global_step2:JPeÿ’:cÔb^ ;DeviceStatsMonitor.on_validation_batch_start/max_split_size"trainer/global_step2:JPƒX>sÕbn KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak"trainer/global_step2:JPý–ZIeÖb` =DeviceStatsMonitor.on_validation_batch_start/segment.all.peak"trainer/global_step2:JPP+ú„m×bh EDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed"trainer/global_step2:JPê…cJpØbk HDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated"trainer/global_step2:JP¿ñY_iÙbd ADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed"trainer/global_step2:JPxóÍtÚbo LDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated"trainer/global_step2:JP4˜>#qÛbl IDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated"trainer/global_step2:JPxª¶T|Übw TDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current"trainer/global_step2:JPòôgqÝbl IDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak"trainer/global_step2:JP -¦ÝtÞbo LDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current"trainer/global_step2:JPyº@ußbp MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed"trainer/global_step2:JPêüÞßmàbh EDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated"trainer/global_step2:JPüBúrábm JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated"trainer/global_step2:JP·KÝjâbe BDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak"trainer/global_step2:JPÞ&zãbu RDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed"trainer/global_step2:JPb—õMräbm JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed"trainer/global_step2:JPZ@‰Œ2å- _wandbruntime‚2118  _runtime‚2118 ÝÚQ,*æ¦T \PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚8697400 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 7134511104 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚22348557 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 572522496 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚582 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚22878959 NADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚55686665 YLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚22878959 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚22348958 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 145454592 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 564258816 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 175409664 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 2399036830208 REDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚55706592 ]JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚49070171746008 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 6966133760 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 2443248058368 ]JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚49027056604160 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 dQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚46677761296208 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚309 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚19927 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1300 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚19925 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚30508 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚49027074607104 dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚49027074607104 `MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚49027056604160 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚55686665 QDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚32807708 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 564258816 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚582 ZGDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚51426286846976 VCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚51426093434368 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚36 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 2443393512960 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚19927 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 6236507136 cQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚ 2392410449800 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 6966133760 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 572522496 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3532 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 UHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚32827633 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚401 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚13651556 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 339145216 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 171817196 XIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 7134511104 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3532 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚30751 XLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚8697001 iVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚51942453085696 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 2392238632604 _step‚96 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚273 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚273 YLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚32827633 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚309 UHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚32807708 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚51942429145600 BDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 560281229 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 188856556 YFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚51426093434368 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚13651558 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 169394688 ^KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚54385677203968 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 2399036830208 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚19925 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚582 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 6741051909 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚582 % _timestamp‚1.7885228800195007e+09 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 bODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚54385846598656 NADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚55706592 ]JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚51426286846976 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 175409664 UHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚22878957 `MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚46677744256848 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚273 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 2399212239872 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1319I/cÃ0*çªT QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚30508 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 193412608 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚1208 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚22878957 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 560281229 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 188856556 YFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚51426093434368 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚13651558 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 169394688 ^KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚54385677203968 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 2399036830208 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚19925 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚582 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 6741051909 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚582 % _timestamp‚1.7885228800195007e+09 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 bODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚54385846598656 NADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚55706592 ]JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚51426286846976 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 175409664 UHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚22878957 `MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚46677744256848 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚273 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 2399212239872 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1319 \PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚8697400 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 7134511104 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚22348557 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 572522496 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚582 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚22878959 NADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚55686665 YLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚22878959 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚22348958 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 145454592 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 564258816 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 175409664 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 2399036830208 REDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚55706592 ]JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚49070171746008 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 6966133760 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 2443248058368 ]JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚49027056604160 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 dQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚46677761296208 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚309 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚19927 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1300 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚19925 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚30508 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚49027074607104 dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚49027074607104 `MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚49027056604160 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚55686665 QDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚32807708 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 564258816 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚582 ZGDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚51426286846976 VCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚51426093434368 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚36 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 2443393512960 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚19927 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 6236507136 cQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚ 2392410449800 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 6966133760 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 572522496 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3532 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 UHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚32827633 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚401 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚13651556 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 339145216 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 171817196 XIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 7134511104 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3532 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚30751 XLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚8697001 iVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚51942453085696 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 2392238632604 _step‚96 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚273 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚273 YLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚32827633 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚309 UHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚32807708 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚51942429145600 Bmærvèbq NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated"trainer/global_step2:JPõ‘ìnébi FDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated"trainer/global_step2:JPs{¥ jêbe BDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak"trainer/global_step2:JPÔtê—rëbm JDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak"trainer/global_step2:JPÚDòçdìb_ GÇsõbn KDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current"trainer/global_step2:JP¦Rlöbg DDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current"trainer/global_step2:JPîëvl÷bg DDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current"trainer/global_step2:JP*àÙjøbe BDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed"trainer/global_step2:JP$Y9nùbi FDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed"trainer/global_step2:JPö0GPzúbu RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current"trainer/global_step2:JP í vûbq NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated"trainer/global_step2:JP¾méÚtübo LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated"trainer/global_step2:JPõ›gcýb^ ;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak"trainer/global_step2:JPmP‡÷iþbd ADeviceStatsMonitor.on_validation_batch_end/allocation.all.current"trainer/global_step2:JPG ¯lÿbg DDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak"trainer/global_step2:JPúÀöw€br ODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated"trainer/global_step2:JPE[j»qbl IDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak"trainer/global_step2:JPH@‰bp‚bk HDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak"trainer/global_step2:JP«Õ—ëcƒb^ ;DeviceStatsMonitor.on_validation_batch_end/active.all.freed"trainer/global_step2:JPkç9us„bn KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed"trainer/global_step2:JPw~k…bf CDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak"trainer/global_step2:JP £‡s†bn KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed"trainer/global_step2:JPõ­s»v‡bq NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated"trainer/global_step2:JP8þñÂxˆbs PDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed"trainer/global_step2:JP%¾!q‰bl IDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak"trainer/global_step2:JPè‹ÃHrŠbm JDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed"trainer/global_step2:JP‰ÈŸ$t‹bo LDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current"trainer/global_step2:JPÝÕFÖeŒb` =DeviceStatsMonitor.on_validation_batch_end/active.all.current"trainer/global_step2:JPa¯~jbe BDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed"trainer/global_step2:JP¦‚jŽbe BDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak"trainer/global_step2:JPŦkbf CDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed"trainer/global_step2:JPøœýmbh EDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak"trainer/global_step2:JPˆ$á´r‘bm JDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed"trainer/global_step2:JPôcŠq’bl IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed"trainer/global_step2:JP×ÐúÔi“bd ADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak"trainer/global_step2:JPÃ[õ9h”bc @DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated"trainer/global_step2:JP-·‹(f•ba >DeviceStatsMonitor.on_validation_batch_end/segment.all.current"trainer/global_step2:JP슊jh–bc @DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak"trainer/global_step2:JP>P1r—bm JDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated"trainer/global_step2:JP¹fÙq˜bl IDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak"trainer/global_step2:JP‹ñÎm™bh EDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current"trainer/global_step2:JPÆo¶Fršbm JDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current"trainer/global_step2:JPÎpâ±m›bh EDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current"trainer/global_step2:JPˆ .ö|œbw TDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated"trainer/global_step2:JPí(žójbe BDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed"trainer/global_step2:JP^z\}lžbg DDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed"trainer/global_step2:JPþ½À…nŸbi FDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated"trainer/global_step2:JPب¹u bp MDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated"trainer/global_step2:JP‡jêžq¡bl IDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated"trainer/global_step2:JP§•ïMn¢bi FDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current"trainer/global_step2:JPal°cj£be BDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak"trainer/global_step2:JP•ž'p¤bk HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed"trainer/global_step2:JPoélˆu¥bp MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current"trainer/global_step2:JP?Ãâªm¦bh EDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak"trainer/global_step2:JPù£Ne§b` =DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections"trainer/global_step2:JP~qv‘i¨bd ADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak"trainer/global_step2:JP.„åÑw©br ODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated"trainer/global_step2:JP ó}Kpªbk HDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current"trainer/global_step2:JPª^no«bj GDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated"trainer/global_step2:JPÉõÁy[¬bV 3DeviceStatsMonitor.on_validation_batch_end/num_ooms"trainer/global_step2:JPÄŒ„k­bf CDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed"trainer/global_step2:JP€‰„ºw®br ODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated"trainer/global_step2:JPþn¢ÿo¯bj GDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current"trainer/global_step2:JP§)S°bf CDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed"trainerýG9}/global_step2:JPú|ýYk±bf CDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak"trainer/global_step2:JPð’n²bi FDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated"trainer/global_step2:JPŒLUi³bd ADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed"trainer/global_step2:JPjï̶o´bj GDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak"trainer/global_step2:JPh€½tµbo LDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current"trainer/global_step2:JPÌWÅs¶bn KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed"trainer/global_step2:JPaÔ×r·bm JDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak"trainer/global_step2:JPPÒB¾o¸bj GDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated"trainer/global_step2:JP Æ<²g¹bb ?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated"trainer/global_step2:JP¢ÂÔ,nºbi FDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current"trainer/global_step2:JPÙèŠf»ba >DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak"trainer/global_step2:JP—^hZb¼b] :DeviceStatsMonitor.on_validation_batch_end/num_device_free"trainer/global_step2:JPMQ6¥l½bg DDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current"trainer/global_step2:JPxžÏ×p¾bk HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated"trainer/global_step2:JP¦*„u¿bp MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current"trainer/global_step2:JP+ùÝ®mÀbh EDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current"trainer/global_step2:JPÈèûÍtÁbo LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated"trainer/global_step2:JPƸÞ|Âbw TDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated"trainer/global_step2:JPA€rÃbm JDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak"trainer/global_step2:JPã`rÄbm JDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed"trainer/global_step2:JPd¤åiÅbd ADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak"trainer/global_step2:JPøÚYÞnÆbi FDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed"trainer/global_step2:JPn=€kÇbf CDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed"trainer/global_step2:JP‘¶xÈbs PDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed"trainer/global_step2:JP…†ÈwÉbr ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak"trainer/global_step2:JPýÿÊngÊbb ?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed"trainer/global_step2:JPÿ«ß£aËb\ 9DeviceStatsMonitor.on_validation_batch_end/max_split_size"trainer/global_step2:JP·ÐPtÌbo LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current"trainer/global_step2:JP ð rÍbm JDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak"trainer/global_step2:JPóLŽªjÎbe BDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak"trainer/global_step2:JPrð™ oÏbj GDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated"trainer/global_step2:JP"p†oÐbj GDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated"trainer/global_step2:JP“amÑbh EDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated"trainer/global_step2:JPªËdpÒbk HDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current"trainer/global_step2:JPô˼wÓbr ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak"trainer/global_step2:JP"ìŸÏcÔb^ ;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc"trainer/global_step2:JP©ù‡kÕbf CDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current"trainer/global_step2:JPöÏG¢pÖbk HDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed"trainer/global_step2:JP¸ðý´p×bk HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated"trainer/global_step2:JP"–d)rØbm JDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed"trainer/global_step2:JP¢nÈÝzÙbu RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current"trainer/global_step2:JPá%ÌgÚbb ?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams"trainer/global_step2:JP}xmÛbh EDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed"trainer/global_step2:JPB$Ã9wÜbr ODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated"trainer/global_step2:JPÎÁÔDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚582 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 6966133760 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚22894341 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚36 TEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 7134511104 trainer/global_step‚1208 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 175409664 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚401 gTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚51972814839296 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 WDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚49096014746896 SFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚32836541 `MDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚54416682212352 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 _runtime‚ 2119.36413854 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 188856556 QBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 7134511104 [HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚49057122278400 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚30508 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3532 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 2399686099968 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚19925 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚22366215 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 bODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚49057140281344 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 6966133760 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚22894341 TADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚51456632968704 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 6458321920 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚399 ^KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚46703303906480 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 560281229 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚309 L?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚55730882 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 193412608 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚30751 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 [HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚49096203603452 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚309 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 2399686099968 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 2443867373056 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 6236507136 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0‘ݯû8)å²R PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚19927 OBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚32816616 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1319 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3532 VJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚8702576 \IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚54416512817664 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚582 F>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚582 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 6966133760 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚22894341 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚36 TEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 7134511104 trainer/global_step‚1208 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 175409664 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚401 gTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚51972814839296 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 WDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚49096014746896 SFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚32836541 `MDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚54416682212352 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 _runtime‚ 2119.36413854 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 188856556 QBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 7134511104 [HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚49057122278400 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚30508 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3532 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 2399686099968 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚19925 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚22366215 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 bODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚49057140281344 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 6966133760 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚22894341 TADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚51456632968704 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 6458321920 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚399 ^KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚46703303906480 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 560281229 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚309 L?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚55730882 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 193412608 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚30751 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 [HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚49096203603452 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚309 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 2399686099968 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 2443867373056 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 6236507136 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚30508 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚22894339 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚22365814 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 2443721918464 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 L?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚55710955 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 6561988608 XJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 564258816 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚309 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚273 $ _timestamp‚1.788522881143532e+09 VGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 7134511104 XEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚51456826381312 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 C;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚582 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 193412608 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 2399510690304 [HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚51456826381312 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚13663238 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 145454592 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 bODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚46703320945840 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 WDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚51456632968704 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 572522496 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 2392710840416 BJå bE "Purity/2FRt1SRqqt0FBt/event_purity"trainer/global_step2:JP>X›@Næ bI &Purity/2FRt4SRqqt0FBt/event_proportion"trainer/global_step2:JPjþèg;ç b6 jet/accuracy_1_of_5"trainer/global_step2:JPȨ²Jè bE "Purity/2FRt3SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP¥K£æJé bE "Purity/4FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP7LhuJê bE "Purity/1FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP¥àÊÃNë bI &Purity/1FRt4SRqqt0FBt/event_proportion"trainer/global_step2:JPt(Nì bI &Purity/3FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JPß/8ÜNí bI &Purity/4FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JP.w²ÈHî bC Purity/3FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPjåùæJï bE "Purity/0FRt2SRqqt3FBt/event_purity"trainer/global_step2:JP9ùÎhJð bE "Purity/0FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP J)ÅHñ bC Purity/1FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JPÇ&¿Hò bC Purity/2FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JP¸rãßNó bI &Purity/4FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JP¦7â{Jô bE "Purity/4FRt0SRqqt0FBt/event_purity"trainer/global_step2:JPÕ=­ÀJõ bE "Purity/*FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP‡›(€Jö bE "Purity/*FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPÕröüN÷ bI &Purity/1FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JP@ã!Nø bI &Purity/1FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JP<¶pHù bC Purity/2FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JP¸[›®Jú bE "Purity/3FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPºtÃJû bE "Purity/1FRt2SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPñâ|ØJü bE "Purity/*FRt3SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPEê ØJý bE "Purity/0FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPµÃ¯0Nþ bI &Purity/1FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPIJ˜òJÿ bE "Purity/1FRt1SRqqt2FBt/event_purity"trainer/global_step2:JPŸNUH€ bC Purity/2FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JPŽŸÏÖN bI &Purity/3FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JP.Mg§J‚ bE "Purity/*FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPQ&ÓNƒ bI &Purity/*FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JPµp´ÉN„ bI &Purity/0FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JPA“xN… bI &Purity/2FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JP6‹¦YN† bI &Purity/3FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JPJ¦&wJ‡ bE "Purity/1FRt1SRqqt4FBt/SRqqt_purity"trainer/global_step2:JPè¨ýiJˆ bE "Purity/*FRt*SRqqt2FBt/event_purity"trainer/global_step2:JP4í íN‰ bI &Purity/*FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JP‰,ÙyNŠ bI &Purity/*FRt4SRqqt0FBt/event_proportion"trainer/global_step2:JPaþèqJ‹ bE "Purity/1FRt3SRqqt1FBt/event_purity"trainer/global_step2:JPfoOJŒ bE "Purity/3FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP]„xN bI &Purity/*FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JP0HiyHŽ bC Purity/0FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JPóõýN bI &Purity/0FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JPÍ_/,H bC Purity/1FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JP’«"9H‘ bC Purity/2FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JPNƒ°&@’ b; particle/accuracy_1_of_1"trainer/global_step2:JPÍqÙH“ bC Purity/3FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JP‰‘­N” bI &Purity/0FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JP.«áN• bI &Purity/2FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JP€{üïN– bI &Purity/4FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JPõÚE˜J— bE "Purity/*FRt4SRqqt1FBt/event_purity"trainer/global_step2:JPǸT{;˜ b6 jet/accuracy_2_of_6"trainer/global_step2:JPS(ÔH™ bC Purity/*FRt0SRqqt4FBt/FRt_purity"trainer/global_step2:JPâ'àìHš bC Purity/*FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JPÒÑfNN› bI &Purity/1FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JP Ђ$Jœ bE "Purity/1FRt3SRqqt*FBt/event_purity"trainer/global_step2:JPQzÁïJ bE "Purity/2FRt1SRqqt1FBt/event_purity"trainer/global_step2:JPõ ‹lHž bC Purity/2FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JP\G»HŸ bC Purity/3FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JP½vJ  bE "Purity/0FRt2SRqqt2FBt/event_purity"trainer/global_step2:JPc© …N¡ bI &Purity/*FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JP¿xßH¢ bC Purity/1FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JPÜjd½J£ bE "Purity/1FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPoÃ!ÔN¤ bI &Purity/1FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JPÉCN¥ bI &Purity/4FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JPˆêµYJ¦ bE "Purity/3FRt4SRqqt*FBt/event_purity"trainer/global_step2:JP¬Æ*H§ bC Purity/*FRt2SRqqt3FBt/FBt_purity"trainer/global_step2:JP™ø)N¨ bI &Purity/4FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JP²¾©9© b4 particle/accuracy"trainer/global_step2:JPe@„šNª bI &Purity/1FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JPjÂ<ÉN« bI &Purity/2FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JP% ²³N¬ bI &Purity/3FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JP:JäŸJ­ bE "Purity/*FRt2SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPV™‚sH® bC Purity/0FRt2SRqqt2FBt/FBt_purity"trainer/global_step2:JPÄJ#J¯ bE "Purity/0FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPGK$N° bI &Purity/*FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JP³ÝØóJ± bE "Purity/*FRt2SRqqt*FBt/event_purity"trainer/global_step2:JPmèe2N² bI &Purity/*FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JPô¢ H³ bC Purity/2FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JP ØäµN´ bI &Purity/3FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JPNµ bI &Purity/3FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JP€»ãN¶ bI &Purity/4FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JP©´àH· bC Purity/*FRt*SRqqt3FBt/FRt_purity"trainer/global_step2:JP(Jº¿H¸ bC Purity/*FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JP!*{J¹ bE "Purity/0FRt*SRqqt*FBt/event_purity"trainer/global_step2:JP7ËY+Hº bC Purity/1FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPkúœqJ» bE "Purity/1FRt3SRqqt0FBt/event_purity"trainer/global_step2:JPmmÌ•N¼ bI &Purity/2FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JPYôWf;½ b6 jet/accuracy_2_of_4"trainer/global_step2:JP“hˆäJ¾ bE "Purity/3FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP¸2µN¿ bI &Purity/*FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JP5IQüHÀ bC Purity/*FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JP™óÆJÁ bE "Purity/*FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPŸ“ÂÚH bC Purity/*FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPv#ø'Nà bI &Purity/1FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JP¢e$HÄ bC Purity/1FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JP‹5ªJÅ bE "Purity/2FRt3SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP¿XÙNÆ bI &Purity/3FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JPµUNÇ bI &Purity/*FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPÿèìNÈ bI &Purity/*FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JPßÑ™HÉ bC Purity/1FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JPÁHáðHÊ bC Purity/2FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JP©AÄÑNË bI &Purity/2FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JPÍW;4NÌ bI &Purity/2FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JP$AÿNÍ bI &Purity/3FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JPèà‘µHÎ bC Purity/1FRt3SRqqt1FBt/FBt_purity"trainer/global_step2:JP-þSaHÏ bC Purity/0FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JP„²úHÐ bC Purity/1FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JP»_í†HÑ bC Purity/2FRt3SRqqt0FBt/FRt_purity"trainer/global_step2:JP¼äÕXHÒ bC Purity/3FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JPsÀöÍHÓ bC Purity/2FRt4SRqqt*FBt/FRt_purity"trainer/global_step2:JPÒ*½HÔ bC Purity/1FRt*SRqqt4FBt/FRt_purity"trainer/global_step2:JP†qˆsHÕ bC Purity/1FRt0SRqqt4FBt/FRt_purity"trainer/global_step2:JPŒnå®HÖ bC Purity/2FRt*SRqqt3FBt/FBt_purity"trainer/global_step2:JPmx;ëJ× bE "Purity/2FRt*SRqqt1FBt/event_purity"trainer/global_step2:JPv±ÂJØ bE "Purity/4FRt*SRqqt*FBt/event_purity"trainer/global_step2:JPxA±¥HÙ bC Purity/4FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JPHØc\;Ú b6 loss/val_total_loss"trainer/global_step2:JPüæ˜JÛ bE "Purity/0FRt*SRqqt1FBt/event_purity"trainer/global_step2:JP?Í8yJÜ bE "Purity/2FRt0SRqqt2FBt/event_purity"trainer/global_step2:JPp œJÝ bE "Purity/2FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPªºÍJÞ bE "Purity/0FRt3SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPº7 ‹Hß bC Purity/1FRt4SRqqt0FBt/FRt_purity"trainer/global_step2:JP˜4½Jà bE "Purity/1FRt4SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP‹‡zJá bE "Purity/3FRt0SRqqt2FBt/event_purity"trainer/global_step2:JPÁXFJâ bE "Purity/0FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPµóÄHã bC Purity/3FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JPãf8ÑJä bE "Purity/1FRt*SRqqt4FBt/event_purity"trainer/global_step2:JPOl&]Jå bE "Purity/2FRt0SRqqt3FBt/event_purity"trainer/global_step2:JPwLHæ bC Purity/3FRt3SRqqt*FBt/FRt_purity"trainer/global_step2:JP _.ÙHç bC Purity/0FRt*SRqqt4FBt/FBt_purity"trainer/global_step2:JP³ÖÍ…Jè bE "Purity/3FRt4SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP[RœBHé bC Purity/2FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JP)+ÄÄ@ê b; particle/accuracy_5_of_7"trainer/global_step2:JPÉEmmJë bE "Purity/*FRt2SRqqt3FBt/SRqqt_purity"trainer/global_step2:JP×ÔANì bI &Purity/*FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JP¤ËxNí bI &Purity/1FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JPj²&@î b; particle/accuracy_2_of_5"trainer/global_step2:JPµî<ï b7 particle/specificity"trainer/global_step2:JP¤îa_;ð b6 jet/accuracy_1_of_7"trainer/global_step2:JPÆác–Nñ bI &Purity/1FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JP¾ìHò bC Purity/3FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JPì!ÃÄJó bE "Purity/4FRt2SRqqt*FBt/event_purity"trainer/global_step2:JP‰ð‡¦Jô bE "Purity/*FRt3SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPÇ.!Jõ bE "Purity/1FRt0SRqqt2FBt/event_purity"trainer/global_step2:JP×&[Hö bC Purity/1FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JP[ñTH÷ bC Purity/2FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JPSï#îJø bE "Purity/2FRt*SRqqt0FBt/event_purity"trainer/global_step2:JPŒ¿Nù bI &Purity/4FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JPÕíjNú bI &Purity/1FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JP¯ ÖZ;û b6 jet/accuracy_5_of_7"trainer/global_step2:JPø/½œJü bE "Purity/4FRt1SRqqt0FBt/event_purity"trainer/global_step2:JP—V¥!Ný bI &Purity/*FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JP%€& Nþ bI &Purity/0FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JPæÌ÷Nÿ bI &Purity/2FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JP ì²3N€ bI &Purity/2FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JPåxNJ bE "Purity/*FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPŽA“YH‚ bC Purity/*FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JPV>êðJƒ bE "Purity/*FRt3SRqqt*FBt/event_purity"trainer/global_step2:JP¨$ö N„ bI &Purity/4FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JPŠ4/ø;… b6 jet/accuracy_2_of_5"trainer/global_step2:JP‡‡ñ8† b3 particle/f_score"trainer/global_step2:JPœ(êïH‹ bC Purity/*FRt4SRqqt*FBt/FBt_purity"trainer/global_step2:JP'@Œ b; particle/accuracy_1_of_7"trainer/global_step2:JPL0¯OJ bE "Purity/3FRt3SRqqt*FBt/event_purity"trainer/global_step2:JPx&JŽ bE "Purity/*FRt3SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP˜s (N bI &Purity/0FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JP™ò°rJ bE "Purity/1FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP‡ú N‘ bI &Purity/1FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JPòEN’ bI &Purity/2FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JP¹°ƒæJ“ bE "Purity/2FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP˜PYzJ” bE "Purity/*FRt0SRqqt4FBt/event_purity"trainer/global_step2:JPî˺ÜJ• bE "Purity/0FRt1SRqqt1FBt/event_purity"trainer/global_step2:JPwØuCN– bI &Purity/2FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JPƒÛ 5H— bC Purity/0FRt*SRqqt3FBt/FBt_purity"trainer/global_step2:JPNK¡H˜ bC Purity/3FRt4SRqqt0FBt/FRt_purity"trainer/global_step2:JPJg¢;™ b6 jet/accuracy_7_of_7"trainer/global_step2:JPh„ùþHš bC Purity/*FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JPr„_¤H› bC Purity/0FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPðïNœ bI &Purity/0FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JPዱ`N bI &Purity/2FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JP˜öÁNž bI &Purity/3FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPUèÒNŸ bI &Purity/4FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JP7 ¾¤;  b6 jet/accuracy_1_of_3"trainer/global_step2:JP=w®H¡ bC Purity/*FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JP¢?J¢ bE "Purity/2FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPTúöâN£ bI &Purity/3FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JPgÄLËN¤ bI &Purity/3FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JPrkj”N¥ bI &Purity/4FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JP·&îJ¦ bE "Purity/2FRt3SRqqt1FBt/event_purity"trainer/global_step2:JPàÑû]J§ bE "Purity/*FRt1SRqqt0FBt/event_purity"trainer/global_step2:JPÍÄ.†H¨ bC Purity/1FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JPÛvGJ© bE "Purity/2FRt3SRqqt0FBt/event_purity"trainer/global_step2:JPÎózmHª bC Purity/3FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JPö‹IàH« bC Purity/3FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JP~†‘H¬ bC Purity/0FRt3SRqqt1FBt/FBt_purity"trainer/global_step2:JPË6–ÇH­ bC Purity/0FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JPš6š©N® bI &Purity/0FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JP²”²N¯ bI &Purity/2FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JPq °YN° bI &Purity/2FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JPš²–N± bI &Purity/3FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JPߤTfJ² bE "Purity/0FRt4SRqqt*FBt/event_purity"trainer/global_step2:JP"ሆ@³ b; particle/accuracy_4_of_6"trainer/global_step2:JP ÍJ´ bE "Purity/4FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPˆ¹ªHµ bC Purity/*FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JPt—ÄèN¶ bI &Purity/1FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JPsª–”H· bC Purity/2FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JP]»è¦J¸ bE "Purity/3FRt*SRqqt*FBt/event_purity"trainer/global_step2:JPPâ«aN¹ bI &Purity/3FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JPºäÓ´Nº bI &Purity/3FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JPg2à!H» bC Purity/2FRt3SRqqt1FBt/FRt_purity"trainer/global_step2:JPà%DH¼ bC Purity/3FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JP¼Ê·ÑH½ bC Purity/*FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JP…S¡CN¾ bI &Purity/1FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JP!–xN¿ bI &Purity/1FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JPÜm4HÀ bC Purity/1FRt3SRqqt*FBt/FRt_purity"trainer/global_step2:JP¯Ø2JÁ bE "Purity/*FRt*SRqqt3FBt/SRqqt_purity"trainer/global_step2:JPO€KH bC Purity/1FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JP¶¡æpNà bI &Purity/2FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JPáì;Ä b6 jet/accuracy_4_of_4"trainer/global_step2:JPUÈäJÅ bE "Purity/1FRt2SRqqt*FBt/event_purity"trainer/global_step2:JPs*1@Æ b; particle/accuracy_5_of_6"trainer/global_step2:JPí¬{JÇ bE "Purity/*FRt2SRqqt2FBt/event_purity"trainer/global_step2:JP–½iJÈ bE "Purity/1FRt*SRqqt3FBt/SRqqt_purity"trainer/global_step2:JPmæw}JÉ bE "Purity/0FRt*SRqqt0FBt/event_purity"trainer/global_step2:JP@dHÊ bC Purity/0FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JPÇjôéJË bE "Purity/1FRt2SRqqt1FBt/event_purity"trainer/global_step2:JP¡_xJÌ bE "Purity/2FRt0SRqqt0FBt/event_purity"trainer/global_step2:JPû>]—HÍ bC Purity/2FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JP¤|³›JÎ bE "Purity/3FRt1SRqqt0FBt/event_purity"trainer/global_step2:JPY‚WôHÏ bC Purity/2FRt3SRqqt*FBt/FBt_purity"trainer/global_step2:JPeœµwHÐ bC Purity/*FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JPÜ]Ô•JÑ bE "Purity/1FRt3SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPŽ{ˆNÒ bI &Purity/2FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JP,f—ENÓ bI &Purity/3FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JP¬âN¢JÔ bE "Purity/3FRt2SRqqt1FBt/event_purity"trainer/global_step2:JPàÆiHÕ bC Purity/3FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JP4¢OMJÖ bE "Purity/3FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP#»­­H× bC Purity/0FRt1SRqqt3FBt/FBt_purity"trainer/global_step2:JPKþ±ŠNØ bI &Purity/*FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JP´eì_HÙ bC Purity/1FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JP8¶)‹JÚ bE "Purity/2FRt*SRqqt*FBt/event_purity"trainer/global_step2:JPvîçHÛ bC Purity/2FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JP$éÙ•NÜ bI &Purity/2FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JP¹äž‘NÝ bI &Purity/2FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JPƒTäwHÞ bC Purity/1FRt3SRqqt2FBt/FRt_purity"trainer/global_step2:JP»,×úHß bC Purity/1FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JP}Ùå½Nà bI &Purity/2FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JP+5>Há bC Purity/3FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JP áÖJâ bE "Purity/0FRt2SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPíUˆHJã bE "Purity/*FRt1SRqqt3FBt/event_purity"trainer/global_step2:JPtbáJä bE "Purity/0FRt*SRqqt3FBt/SRqqt_purity"trainer/global_step2:JPd[óNå bI &Purity/*FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JPã¤OiNæ bI &Purity/0FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPª ì Hç bC Purity/0FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JPY™ßÃNè bI &Purity/0FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JP ÃŒþJé bE "Purity/1FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPjÆpbJê bE "Purity/1FRt2SRqqt0FBt/event_purity"trainer/global_step2:JP)#ËÃNë bI &Purity/3FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JPÕâšÓJì bE "Purity/*FRt*SRqqt4FBt/event_purity"trainer/global_step2:JPûQ£ŒJí bE "Purity/*FRt0SRqqt0FBt/event_purity"trainer/global_step2:JPç¸Jî bE "Purity/0FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPœ;€Hï bC Purity/2FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JPPÏanNð bI &Purity/4FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JPFANñ bI &Purity/4FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JPþùJò bE "Purity/1FRt2SRqqt2FBt/event_purity"trainer/global_step2:JPtcçŸHó bC Purity/2FRt2SRqqt2FBt/FBt_purity"trainer/global_step2:JPz,׃Jô bE "Purity/*FRt0SRqqt3FBt/event_purity"trainer/global_step2:JP_h†Jõ bE "Purity/*FRt3SRqqt0FBt/event_purity"trainer/global_step2:JPj8b/Jö bE "Purity/1FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP†²èH÷ bC Purity/2FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JP!7½Jø bE "Purity/1FRt4SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP°ý^Jù bE "Purity/2FRt2SRqqt*FBt/event_purity"trainer/global_step2:JPÂS!lJú bE "Purity/3FRt0SRqqt*FBt/event_purity"trainer/global_step2:JPŒ^ãÕJû bE "Purity/*FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPR9è£Nü bI &Purity/*FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JPUÌNý bI &Purity/1FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JPãg-Jþ bE "Purity/3FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP™óJÿ bE "Purity/3FRt*SRqqt2FBt/event_purity"trainer/global_step2:JPjJ€ bE "Purity/*FRt0SRqqt1FBt/event_purity"trainer/global_step2:JP(.} N bI &Purity/*FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JP J‚ bE "Purity/1FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP°ÜŒOJƒ bE "Purity/1FRt1SRqqt0FBt/event_purity"trainer/global_step2:JP*âl²J„ bE "Purity/2FRt1SRqqt2FBt/event_purity"trainer/global_step2:JPoª>ýN… bI &Purity/4FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JP"Â@† b; particle/accuracy_4_of_7"trainer/global_step2:JPCKH‡ bC Purity/*FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JP²?ÒÑHˆ bC Purity/0FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JP¤ÈsðN‰ bI &Purity/4FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPÍ×KNŠ bI &Purity/4FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JPݤ=ÕN‹ bI &Purity/4FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JPb–HHŒ bC Purity/*FRt2SRqqt2FBt/FBt_purity"trainer/global_step2:JP(Ô]; b6 jet/accuracy_3_of_7"trainer/global_step2:JP8ßNŽ bI &Purity/*FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JPÄtSãJ bE "Purity/0FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPȘ-H@ b; particle/accuracy_6_of_6"trainer/global_step2:JPí>H‘ bC Purity/1FRt1SRqqt4FBt/FBt_purity"trainer/global_step2:JPIwýJ’ bE "Purity/*FRt2SRqqt1FBt/event_purity"trainer/global_step2:JPû¢ò‹N“ bI &Purity/1FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JPP“´þN” bI &Purity/2FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JPf¸/N• bI &Purity/4FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JPè1JöN– bI &Purity/4FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JPvŽwþH— bC Purity/*FRt4SRqqt0FBt/FRt_purity"trainer/global_step2:JP|Ò8H˜ bC Purity/1FRt2SRqqt2FBt/FRt_purity"trainer/global_step2:JP)wFJ™ bE "Purity/0FRt2SRqqt3FBt/SRqqt_purity"trainer/global_step2:JPàë)¹Hš bC Purity/*FRt3SRqqt*FBt/FRt_purity"trainer/global_step2:JPáçžÉJ› bE "Purity/4FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP$²Nœ bI &Purity/*FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JP_˜„N bI &Purity/0FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JPe÷øLJž bE "Purity/1FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPÿ9‡•;Ÿ b6 jet/accuracy_2_of_2"trainer/global_step2:JP?âÿH  bC Purity/*FRt1SRqqt4FBt/FBt_purity"trainer/global_step2:JPiþUÉH¡ bC Purity/4FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JPÊÖ÷J¢ bE "Purity/*FRt*SRqqt0FBt/event_purity"trainer/global_step2:JP·_èH£ bC Purity/*FRt0SRqqt3FBt/FRt_purity"trainer/global_step2:JPähøÆH¤ bC Purity/*FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JPÒ˜&óN¥ bI &Purity/*FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JP8Àò·H¦ bC Purity/4FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JPl›÷yH§ bC Purity/4FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JP5zLÁN¨ bI &Purity/0FRt4SRqqt0FBt/event_proportion"trainer/global_step2:JPÞ ÞÊN© bI &Purity/4FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JPÕŽFÒNª bI &Purity/4FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JP{ˆîN« bI &Purity/4FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JPC×J¬ bE "Purity/3FRt0SRqqt1FBt/event_purity"trainer/global_step2:JP°RsDH­ bC Purity/0FRt0SRqqt4FBt/FBt_purity"trainer/global_step2:JPùÐzmJ® bE "Purity/*FRt3SRqqt2FBt/event_purity"trainer/global_step2:JPáÒ†ÀN¯ bI &Purity/0FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JP'3ÕåH° bC Purity/2FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JPíÒFN± bI &Purity/2FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JPÉû ;² b6 jet/accuracy_3_of_3"trainer/global_step2:JP íìH³ bC Purity/0FRt0SRqqt3FBt/FBt_purity"trainer/global_step2:JPyòqM'´ b6 jet/accuracy_6_of_7"trainer/gloYìØbal_step2:JP¦†3Hµ bC Purity/*FRt*SRqqt4FBt/FRt_purity"trainer/global_step2:JP­ÍôJ¶ bE "Purity/1FRt1SRqqt3FBt/event_purity"trainer/global_step2:JP”/áH· bC Purity/1FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JPšEëÆJ¸ bE "Purity/2FRt0SRqqt1FBt/event_purity"trainer/global_step2:JPºzH¹ bC Purity/3FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JP ²uNº bI &Purity/4FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JP‚ƈ\N» bI &Purity/*FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JPÇŽÑN¼ bI &Purity/3FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JPL¸¡ @½ b; particle/accuracy_1_of_5"trainer/global_step2:JP=ƒ±B;¾ b6 jet/accuracy_6_of_6"trainer/global_step2:JPB“EH¿ bC Purity/*FRt1SRqqt3FBt/FBt_purity"trainer/global_step2:JPQúJ¿HÀ bC Purity/*FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JPYñNÁ bI &Purity/*FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JPÁ†N bI &Purity/*FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JPÈ‚ËVJà bE "Purity/2FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP HÄ bC Purity/2FRt0SRqqt3FBt/FBt_purity"trainer/global_step2:JP”*NÅ bI &Purity/*FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JPódJÆ bE "Purity/2FRt0SRqqt*FBt/event_purity"trainer/global_step2:JPa›cµJÇ bE "Purity/2FRt3SRqqt*FBt/event_purity"trainer/global_step2:JP a> JÈ bE "Purity/0FRt*SRqqt4FBt/event_purity"trainer/global_step2:JPž¢åHÉ bC Purity/0FRt4SRqqt*FBt/FBt_purity"trainer/global_step2:JPÿ¬5JÊ bE "Purity/0FRt3SRqqt1FBt/event_purity"trainer/global_step2:JPª­iÜJË bE "Purity/1FRt4SRqqt0FBt/event_purity"trainer/global_step2:JP¤¢ñaJÌ bE "Purity/*FRt1SRqqt4FBt/SRqqt_purity"trainer/global_step2:JPI½A JÍ bE "Purity/0FRt1SRqqt0FBt/event_purity"trainer/global_step2:JP¡~«NÎ bI &Purity/0FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JP‹f HÏ bC Purity/1FRt*SRqqt3FBt/FRt_purity"trainer/global_step2:JP| @Ð b; particle/accuracy_2_of_7"trainer/global_step2:JP|à*±JÑ bE "Purity/0FRt1SRqqt3FBt/event_purity"trainer/global_step2:JP*²¶HÒ bC Purity/1FRt1SRqqt3FBt/FBt_purity"trainer/global_step2:JP„®? NÓ bI &Purity/0FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JP]søJÔ bE "Purity/1FRt0SRqqt3FBt/event_purity"trainer/global_step2:JP©`¸ÏHÕ bC Purity/1FRt0SRqqt3FBt/FRt_purity"trainer/global_step2:JPKë|JÖ bE "Purity/1FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPè‚»J× bE "Purity/3FRt1SRqqt1FBt/event_purity"trainer/global_step2:JP.,’-JØ bE "Purity/0FRt4SRqqt1FBt/event_purity"trainer/global_step2:JP‹Ú‰V;Ù b6 jet/accuracy_5_of_6"trainer/global_step2:JPROJÚ bE "Purity/3FRt4SRqqt0FBt/event_purity"trainer/global_step2:JPräc”HÛ bC Purity/2FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JPwÀ÷JÜ bE "Purity/2FRt4SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPcʱG;Ý b6 jet/accuracy_1_of_6"trainer/global_step2:JPýòͲHÞ bC Purity/3FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JPc¯*kHß bC Purity/*FRt3SRqqt2FBt/FRt_purity"trainer/global_step2:JP­ŠWHà bC Purity/0FRt2SRqqt3FBt/FBt_purity"trainer/global_step2:JPvä^±Já bE "Purity/0FRt1SRqqt2FBt/event_purity"trainer/global_step2:JP“¢ uNâ bI &Purity/1FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JPüòNã bI &Purity/2FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JPê ¥Nä bI &Purity/3FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JP­¾~Nå bI &Purity/3FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JPÏf×ãJæ bE "Purity/4FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP@«6ûJç bE "Purity/*FRt2SRqqt3FBt/event_purity"trainer/global_step2:JP 6RCJè bE "Purity/1FRt*SRqqt*FBt/event_purity"trainer/global_step2:JPìþaˆHé bC Purity/1FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JPwÉ´èHê bC Purity/2FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JPcƒ¡yNë bI &Purity/2FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JPòÐ)àNì bI &Purity/3FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JPor¾Jí bE "Purity/0FRt0SRqqt2FBt/event_purity"trainer/global_step2:JPŒ UCJî bE "Purity/0FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPp€DNï bI &Purity/3FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JPÙ«¾Jð bE "Purity/3FRt1SRqqt2FBt/event_purity"trainer/global_step2:JP9Z7Hñ bC Purity/1FRt4SRqqt*FBt/FRt_purity"trainer/global_step2:JP•êþËJò bE "Purity/1FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPxgˆDNó bI &Purity/2FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JPÙ·]þNô bI &Purity/4FRt4SRqqt0FBt/event_proportion"trainer/global_step2:JP°ìÁ`Hõ bC Purity/3FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JPÕfqNö bI &Purity/*FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JP¡š7ÉN÷ bI &Purity/*FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JPìh4cNø bI &Purity/2FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JP Ïå©;ù b6 jet/accuracy_3_of_5"trainer/global_step2:JP¾ŽVJú bE "Purity/3FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPçëá@û b; particle/accuracy_3_of_7"trainer/global_step2:JPÆp,–Jü bE "Purity/4FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP–7åNý bI &Purity/0FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JPv^vVNþ bI &Purity/0FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JP¼wùHÿ bC Purity/1FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JP© 7@€ b; particle/accuracy_3_of_5"trainer/global_step2:JP/8 @ b; particle/accuracy_3_of_6"trainer/global_step2:JP GâgJ‚ bE "Purity/2FRt2SRqqt2FBt/event_purity"trainer/global_step2:JPd¯ÏÿNƒ bI &Purity/0FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JPu=ÙN„ bI &Purity/1FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JPXì#ªJ… bE "Purity/3FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPS×'N† bI &Purity/3FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JPìÛD“J‡ bE "Purity/2FRt*SRqqt3FBt/event_purity"trainer/global_step2:JPõò[Nˆ bI &Purity/2FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JP¥Ø¹¬@‰ b; particle/accuracy_2_of_6"trainer/global_step2:JP]%Zö;Š b6 jet/accuracy_2_of_7"trainer/global_step2:JPaËeñH‹ bC Purity/*FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JP­ÇšDJŒ bE "Purity/2FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPϨz8N bI &Purity/3FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JPü þÿHŽ bC Purity/*FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JP”õþ®N bI &Purity/0FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JP¬OúgN bI &Purity/1FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JP0÷ç^N‘ bI &Purity/3FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JPPº;’ b6 jet/accuracy_1_of_4"trainer/global_step2:JPÔ#wÿ@“ b; particle/accuracy_4_of_5"trainer/global_step2:JPºcã+H” bC Purity/*FRt3SRqqt1FBt/FRt_purity"trainer/global_step2:JPÊå2]@• b; particle/accuracy_7_of_7"trainer/global_step2:JPzÍþN– bI &Purity/3FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JPxÅÒ1N— bI &Purity/4FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JP6< 9J˜ bE "Purity/*FRt*SRqqt4FBt/SRqqt_purity"trainer/global_step2:JP²Ê7J™ bE "Purity/4FRt0SRqqt*FBt/event_purity"trainer/global_step2:JP¿B}Nš bI &Purity/0FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JPÊãTˆJ› bE "Purity/2FRt3SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPN;æJœ bE "Purity/3FRt*SRqqt1FBt/event_purity"trainer/global_step2:JPø‚J bE "Purity/2FRt4SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPl û!Jž bE "Purity/0FRt3SRqqt0FBt/event_purity"trainer/global_step2:JP ‘qïHŸ bC Purity/*FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JPD J  bE "Purity/0FRt0SRqqt*FBt/event_purity"trainer/global_step2:JP–OÁîN¡ bI &Purity/1FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JP:ìÃH¢ bC Purity/*FRt4SRqqt1FBt/FBt_purity"trainer/global_step2:JPÝò«ñJ£ bE "Purity/2FRt4SRqqt*FBt/event_purity"trainer/global_step2:JPH¼H¤ bC Purity/3FRt4SRqqt*FBt/FRt_purity"trainer/global_step2:JPÕ÷Ê‚@¥ b; particle/accuracy_2_of_4"trainer/global_step2:JPSnÂH¦ bC Purity/*FRt3SRqqt*FBt/FBt_purity"trainer/global_step2:JP“ê<ÚJ§ bE "Purity/0FRt0SRqqt4FBt/event_purity"trainer/global_step2:JPˆ¹rç;¨ b6 jet/accuracy_3_of_6"trainer/global_step2:JPX‚ì–J© bE "Purity/0FRt2SRqqt1FBt/event_purity"trainer/global_step2:JPÛH±Hª bC Purity/1FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JP®8O@N« bI &Purity/4FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JPe;ç2@¬ b; particle/accuracy_1_of_2"trainer/global_step2:JP¬\OJ­ bE "Purity/0FRt4SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPÊÔ^zJ® bE "Purity/4FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP[nýJ¯ bE "Purity/*FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP!ïoH° bC Purity/0FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JP¤)ÎN± bI &Purity/0FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JP¼ÕühJ² bE "Purity/2FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP-XH³ bC Purity/2FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JPþêfH´ bC Purity/3FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JPÇyHµ bC Purity/*FRt1SRqqt4FBt/FRt_purity"trainer/global_step2:JPvæGH¶ bC Purity/*FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JP•&ÄJ· bE "Purity/1FRt*SRqqt0FBt/event_purity"trainer/global_step2:JP›Œ|qN¸ bI &Purity/1FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JP&Ù\\H¹ bC Purity/2FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JPÈâkeNº bI &Purity/2FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JPü•¥°N» bI &Purity/4FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JPZâtH¼ bC Purity/*FRt*SRqqt3FBt/FBt_purity"trainer/global_step2:JP6Z½H½ bC Purity/1FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JPhÇ›N¾ bI &Purity/1FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JPÀü·H¿ bC Purity/2FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JPpà§öJÀ bE "Purity/3FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPÄ'fHÁ bC Purity/3FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JP|¡éN bI &Purity/0FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JP£ôÀ»Hà bC Purity/1FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JP"*?JÄ bE "Purity/1FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP0¼ÍÄNÅ bI &Purity/4FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JPÛ¨ÑHÆ bC Purity/1FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JPq\cØHÇ bC Purity/3FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JP“[švJÈ bE "Purity/*FRt1SRqqt2FBt/event_purity"trainer/global_step2:JP46½üJÉ bE "Purity/0FRt*SRqqt2FBt/event_purity"trainer/global_step2:JP3ö,½JÊ bE "Purity/3FRt0SRqqt0FBt/event_purity"trainer/global_step2:JPw9 EHË bC Purity/2FRt*SRqqt3FBt/FRt_purity"trainer/global_step2:JP‡‘ÛTHÌ bC Purity/1FRt1SRqqt4FBt/FRt_purity"trainer/global_step2:JP£’!JÍ bE "Purity/*FRt0SRqqt*FBt/event_purity"trainer/global_step2:JP ºËJÎ bE "Purity/1FRt*SRqqt1FBt/event_purity"trainer/global_step2:JP8î‚HÏ bC Purity/1FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JP©ËáÂNÐ bI &Purity/1FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JPXÖŸVNÑ bI &Purity/1FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JPt÷ã¹JÒ bE "Purity/3FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP þâNÓ bI &Purity/3FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JP«˜ž@Ô b; particle/accuracy_4_of_4"trainer/global_step2:JP(Wí+HÕ bC Purity/*FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JPjSG×HÖ bC Purity/*FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JPLÛT”N× bI &Purity/1FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JP"cî“JØ bE "Purity/2FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP:’ýÌHÙ bC Purity/2FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JP“£eûNÚ bI &Purity/4FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JPÿ¢É7HÛ bC Purity/1FRt3SRqqt*FBt/FBt_purity"trainer/global_step2:JP±¨qÅHÜ bC Purity/3FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JPVÖ±NÝ bI &Purity/*FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JPàçäbNÞ bI &Purity/0FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JPÜV]'Hß bC Purity/2FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JPo¸Jà bE "Purity/3FRt1SRqqt*FBt/event_purity"trainer/global_step2:JP¶Ú¿_Há bC Purity/2FRt4SRqqt0FBt/FRt_purity"trainer/global_step2:JP^ì;¤Hâ bC Purity/3FRt3SRqqt0FBt/FRt_purity"trainer/global_step2:JPž EÚJã bE "Purity/1FRt3SRqqt2FBt/event_purity"trainer/global_step2:JPr2'Nä bI &Purity/*FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JPåyž¥Nå bI &Purity/*FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JP%{Jæ bE "Purity/0FRt0SRqqt0FBt/event_purity"trainer/global_step2:JPÙ“>\Jç bE "Purity/1FRt1SRqqt*FBt/event_purity"trainer/global_step2:JP®ŸÂÀ@è b; particle/accuracy_3_of_4"trainer/global_step2:JPTßà°Jé bE "Purity/1FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPj&7‰Jê bE "Purity/2FRt*SRqqt2FBt/event_purity"trainer/global_step2:JPº9õJë bE "Purity/*FRt4SRqqt0FBt/event_purity"trainer/global_step2:JPß–tIJì bE "Purity/*FRt1SRqqt4FBt/event_purity"trainer/global_step2:JPñÑ8Jí bE "Purity/1FRt1SRqqt1FBt/event_purity"trainer/global_step2:JP80fNî bI &Purity/3FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JPW†mJï bE "Purity/0FRt0SRqqt3FBt/event_purity"trainer/global_step2:JP?tñ­Hð bC Purity/1FRt3SRqqt1FBt/FRt_purity"trainer/global_step2:JPtt¹ÎJñ bE "Purity/3FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPR ÕŸHò bC Purity/*FRt*SRqqt4FBt/FBt_purity"trainer/global_step2:JPr”Jó bE "Purity/0FRt4SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP¨ÙæˆJô bE "Purity/1FRt4SRqqt*FBt/event_purity"trainer/global_step2:JPÛc‡øJõ bE "Purity/0FRt2SRqqt*FBt/event_purity"trainer/global_step2:JPB WûNö bI &Purity/1FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JPVâVH÷ bC Purity/2FRt3SRqqt*FBt/FRt_purity"trainer/global_step2:JPvÆcHø bC Purity/3FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JPÓJù bE "Purity/0FRt4SRqqt0FBt/event_purity"trainer/global_step2:JP Q!aJú bE "Purity/3FRt3SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP_únžNû bI &Purity/0FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JP¸¶£ÌNü bI &Purity/0FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JPÄCãËJý bE "Purity/0FRt3SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPb—eHþ bC Purity/1FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JP?_ Hÿ bC Purity/*FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JP(—GºJ€bE "Purity/*FRt0SRqqt2FBt/event_purity"trainer/global_step2:JP–WEÛJbE "Purity/*FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPžø¾N‚bI &Purity/0FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JP=7JƒbE "Purity/2FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP¥‡VØN„bI &Purity/4FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JPIÝÁH…bC Purity/4FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JPºûýeJ†bE "Purity/*FRt*SRqqt1FBt/event_purity"trainer/global_step2:JPèÙÊN‡bI &Purity/*FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JPÒ·NˆbI &Purity/1FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JPÅ“yšN‰bI &Purity/3FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JP>jÁHŠbC Purity/*FRt4SRqqt*FBt/FRt_purity"trainer/global_step2:JP/ÞÁTH‹bC Purity/3FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JPO!¶E#HšbC Purity/2FRt0SRqqt3FBt/FRt_purity"trainer/global_step2:JPnH›bC Purity/0FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JPqìD³JœbE "Purity/2FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPÅ"RJbE "Purity/3FRt2SRqqt0FBt/event_purity"trainer/global_step2:JPdŸNžbI &Purity/4FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JPnr`nJŸbE "Purity/1FRt0SRqqt4FBt/event_purity"trainer/global_step2:JPÝ)ûJ bE "Purity/4FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPþ×*H¡bC Purity/*FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JPÚ¬RåH¢bC Purity/*FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JPë -ÎH£bC Purity/1FRt*SRqqt3FBt/FBt_purity"trainer/global_step2:JPµñ‡J¤bE "Purity/1FRt0SRqqt0FBt/event_purity"trainer/global_step2:JPk;ÿJ¥bE "Purity/1FRt1SRqqt4FBt/event_purity"trainer/global_step2:JPEæ¾ÌH¦bC Purity/1FRt1SRqqt3FBt/FRt_purity"trainer/global_step2:JP]fÿH§bC Purity/4FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JP¿içH¨bC Purity/4FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JPvXN©bI &Purity/0FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JPº¹SgHªbC Purity/1FRt0SRqqt3FBt/FBt_purity"trainer/global_step2:JPéw˜XH«bC Purity/2FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JP¿RxëJ¬bE "Purity/2FRt2SRqqt0FBt/event_purity"trainer/global_step2:JPÿ?X°N­bI &Purity/2FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JPº ½WH®bC Purity/4FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JPø<ѧH¯bC Purity/*FRt3SRqqt0FBt/FRt_purity"trainer/global_step2:JPLjæ@N°bI &Purity/0FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JPvÓN±bI &Purity/3FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JP[±N²bI &Purity/3FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JPShJ³bE "Purity/0FRt*SRqqt3FBt/event_purity"trainer/global_step2:JPÚPJ´bE "Purity/*FRt4SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP¼´%@µb; particle/accuracy_6_of_7"trainer/global_step2:JP¯Ìk„J¶bE "Purity/*FRt1SRqqt3FBt/SRqqt_purity"trainer/global_step2:JP•6fý2·- _wandbruntime‚2136  _runtime‚2136̶ .¸Õ¥ :"Purity/1FRt0SRqqt2FBt/event_purity‚0.08380171656608582 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9770114421844482 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9951940178871155 ("Purity/2FRt*SRqqt0FBt/event_purity‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 jet/accuracy_5_of_7‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.21441911160945892 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2848984897136688 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 +jet/accuracy_2_of_5‚0.24328595399856567 'particle/f_score‚0.2992917597293854 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.1180555522441864 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.16339637339115143 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_7‚1 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ("Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.26742446422576904 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.5227272510528564 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.0404294990003109 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9225146770477295 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 jet/accuracy_7_of_7‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9878804683685303 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9594109058380127 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.7053794860839844 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.992639422416687 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0022675737272948027 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.12501008808612823 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.2619539797306061 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9849578142166138 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_4_of_6‚1 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9956778287887573 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9920635223388672 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.0833333358168602 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9835266470909119 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.21120689809322357 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.550000011920929 6 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.986067533493042 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 -jet/accuracy_4_of_4‚0.0030423279386013746 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_5_of_6‚1 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 *"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.5 _step‚98 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.1854531317949295 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9877192974090576 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ("Purity/2FRt0SRqqt0FBt/event_purity‚0 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.24828870594501495 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 trainer/global_step‚1208 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.2748599052429199 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.13879141211509705 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.13425925374031067 :"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.35879629850387573 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9523810148239136 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9941313862800598 <"Purity/2FRt*SRqqt*FBt/event_purity‚0.0018479771679267287 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9964672327041626 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.15740744769573212 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.2142857164144516 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.4375 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9789462685585022 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.29763051867485046 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.0708044171333313 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.1723482757806778 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.08596938848495483 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 8"Purity/*FRt0SRqqt3FBt/event_purity‚0.095238097012043 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2743055522441864 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.30223211646080017 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.2083333283662796 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.3222093880176544 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 :"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.35087719559669495 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6182194352149963 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.24642224609851837 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.005916070193052292 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 particle/accuracy_4_of_7‚1 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1614340841770172 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9778561592102051 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9633333683013916 jet/accuracy_3_of_7‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.2704177796840668 particle/accuracy_6_of_6‚1 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.0833333358168602 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.3333333432674408 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.16243843734264374 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.3438422679901123 ,jet/accuracy_2_of_2‚0.047107502818107605 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 & Purity/4FRt1SRqqt*FBt/FRt_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.08593720942735672 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.3529411852359772 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.1287277489900589 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 & Purity/4FRt*SRqqt*FBt/FRt_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,jet/accuracy_3_of_3‚0.009137182496488094 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9184027910232544 jet/accuracy_6_of_7‚0 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.17731748521327972 ;"Purity/2FRt0SRqqt1FBt/event_purity‚0.011904762126505375 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.07726865261793137 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 particle/accuracy_1_of_5‚1 jet/accuracy_6_of_6‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9444444179534912 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.35823625326156616 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.22245068848133087 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.002917833859100938 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 +"Purity/0FRt*SRqqt4FBt/event_purity‚0.25 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚0 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.23806782066822052 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.3611111044883728 particle/accuracy_2_of_7‚1 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.9166666865348816 _runtime‚ 2136.19894201 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.03125 + Purity/1FRt0SRqqt3FBt/FRt_purity‚0.3125 :"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.38411223888397217 ("Purity/3FRt1SRqqt1FBt/event_purity‚˜ÚUáMd0 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 jet/accuracy_5_of_6‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.0985746756196022 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.10000000149011612 *jet/accuracy_1_of_6‚0.8461538553237915 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 & Purity/0FRt2SRqqt3FBt/FBt_purity‚1 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.021551724523305893 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 ;"Purity/1FRt*SRqqt*FBt/event_purity‚0.039341311901807785 6 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.971230149269104 6 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.998236358165741 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.31690213084220886 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.15636730194091797 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.24698275327682495 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.04764464870095253 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.2142857164144516 particle/accuracy_3_of_7‚1 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.24712643027305603 particle/accuracy_3_of_5‚1 particle/accuracy_3_of_6‚1 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.2638888955116272 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_6‚1 jet/accuracy_2_of_7‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9843390583992004 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.19900479912757874 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9911186695098877 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.7662835717201233 particle/accuracy_4_of_5‚1 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.2750000059604645 particle/accuracy_7_of_7‚0.5 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.13123124837875366 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.10000000149011612 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.2772643566131592 7"Purity/0FRt0SRqqt*FBt/event_purity‚0.80561763048172 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 particle/accuracy_2_of_4‚1 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 +"Purity/0FRt0SRqqt4FBt/event_purity‚0.25 ,jet/accuracy_3_of_6‚0.057692307978868484 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9700201153755188 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_1_of_2‚1 ,"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.125 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.25531309843063354 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9904622435569763 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.2676083743572235 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3520282208919525 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.03800273314118385 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.1631327122449875 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9255319237709045 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9645384550094604 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.24122807383537292 :"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.30560699105262756 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.11873605847358704 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.18682323396205902 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.4309028089046478 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2291666567325592 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.009050179272890091 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2447042614221573 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 & Purity/2FRt*SRqqt3FBt/FRt_purity‚1 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.31673479080200195 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.03940335288643837 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.08438822627067566 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.24534553289413452 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 particle/accuracy_4_of_4‚1 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.980311393737793 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.16997353732585907 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.24669522047042847 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.2348484992980957 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.2361111044883728 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.13426841795444489 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1041666716337204 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.005023447331041098 particle/accuracy_3_of_4‚1 :"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.28026556968688965 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.0023547881282866 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.109375 + Purity/1FRt3SRqqt1FBt/FRt_purity‚0.3125 ,"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.375 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.8333333134651184 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.006569664925336838 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.07027027010917664 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.08024691790342331 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.1666666716337204 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.07407407462596893 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.17523148655891418 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.0839441642165184 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.22242550551891327 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.17428256571292877 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.2261432409286499 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.34050101041793823 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.0833333358168602 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.2499999850988388 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.16612200438976288 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.11092990636825562 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.02271396666765213 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.07753463089466095 particle/accuracy_2_of_3‚1 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 jet/accuracy_5_of_5‚0 & Purity/2FRt0SRqqt3FBt/FRt_purity‚1 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9842535853385925 :"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.35505953431129456 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.976729154586792 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.1569526046514511 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.08438822627067566 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 & Purity/4FRt*SRqqt0FBt/FRt_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.2422480434179306 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 & Purity/4FRt1SRqqt0FBt/FRt_purity‚0 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.16408516466617584 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.09210526198148727 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0 particle/accuracy_6_of_7‚1 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.5555555820465088 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 :"Purity/1FRt*SRqqt3FBt/event_purity‚0.02777777798473835 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ,jet/accuracy_3_of_4‚0.025770392268896103 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.23820030689239502 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.15563608705997467 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.11056467890739441 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.1666666716337204 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.15724022686481476 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 particle/accuracy_3_of_3‚1 ,particle/sensitivity‚0.24307286739349365 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.27144908905029297 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.14670683443546295 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9956948161125183 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 :"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.34051743149757385 *jet/accuracy_1_of_2‚0.5933186411857605 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9333333373069763 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9219577312469482 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.2949201464653015 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.08510638028383255 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.007946529425680637 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.34375 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.2619685232639313 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.22602339088916779 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 particle/accuracy_1_of_4‚1 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.4285714328289032 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.09776625782251358 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9935897588729858 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 7validation_average_jet_accuracy‚0.31863293051719666 jet/accuracy_4_of_6‚0 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.095238097012043 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.1428571492433548 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.3076989948749542 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.27587902545928955 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.002889381255954504 5 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.99622642993927 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9956948161125183 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.08253729343414307 particle/accuracy_2_of_2‚1 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 epoch‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 +jet/accuracy_2_of_3‚0.11868831515312195 particle/accuracy_1_of_6‚1 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 *"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.5 ("Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9897347092628479 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 ("Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9852989315986633 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 ;"Purity/1FRt*SRqqt2FBt/event_purity‚0.051590751856565475 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.15325669944286346 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.5075757503509521 7 Purity/2FRt0SRqqt1FBt/FBt_purity‚0.9940476417541504 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.27178969979286194 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 *jet/accuracy_1_of_1‚0.3500213325023651 jet/accuracy_4_of_5‚0 ,"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.125 jet/accuracy_4_of_7‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.10000000149011612 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.22945907711982727 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.3294546902179718 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.2736682593822479 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.18914705514907837 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.2851990759372711 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.07575757056474686 6 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.984627366065979 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 *jet/accuracy_1_of_5‚0.7840117812156677 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.2222222238779068 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.22029633820056915 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 % _timestamp‚1.7885228979748118e+09 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.3296022117137909 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9845783114433289 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.13850151002407074 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.27964451909065247 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.3555610477924347 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 + Purity/2FRt2SRqqt1FBt/FRt_purity‚0.3125 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.2715517282485962 +"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.25 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.15102194249629974 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.1852608323097229 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.1117895171046257 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.3205678164958954 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚0 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.16254638135433197 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9825173020362854 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.16429202258586884 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_1‚1 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 +jet/accuracy_2_of_6‚0.20512820780277252 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.37173545360565186 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 :"Purity/2FRt1SRqqt1FBt/event_purity‚0.01785714365541935 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.17048294842243195 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.17496705055236816 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 & Purity/*FRt2SRqqt3FBt/FBt_purity‚1 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 (particle/accuracy‚0.8222914338111877 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 +"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.25 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9642857313156128 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.2071743607521057 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.002381041878834367 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.08019164949655533 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.3947368562221527 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1268998235464096 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.3379662036895752 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9635416865348816 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 +jet/accuracy_2_of_4‚0.22064168751239777 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.36000001430511475 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.15729494392871857 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.2523239254951477 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9690539836883545 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1356145739555359 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.1067340075969696 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.41027548909187317 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9825683832168579 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.4101613163948059 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.07499999552965164 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.09195402264595032 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.013888888992369175 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 *loss/val_total_loss‚58.641594073572314 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.5926128625869751 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.2966269850730896 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.1662309318780899 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.2083333283662796 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.23806782066822052 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1041666716337204 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 ("Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.2970939874649048 particle/accuracy_5_of_7‚1 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 +particle/specificity‚0.9294711351394653 jet/accuracy_1_of_7‚0 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.20277778804302216 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.185185179114341749é„Ð¥¹Ù¥ 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.17523148655891418 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.0839441642165184 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.22242550551891327 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.17428256571292877 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.2261432409286499 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.34050101041793823 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.0833333358168602 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.2499999850988388 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.16612200438976288 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.11092990636825562 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.02271396666765213 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.07753463089466095 particle/accuracy_2_of_3‚1 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 jet/accuracy_5_of_5‚0 & Purity/2FRt0SRqqt3FBt/FRt_purity‚1 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9842535853385925 :"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.35505953431129456 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.976729154586792 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.1569526046514511 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.08438822627067566 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 & Purity/4FRt*SRqqt0FBt/FRt_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.2422480434179306 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 & Purity/4FRt1SRqqt0FBt/FRt_purity‚0 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.16408516466617584 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.09210526198148727 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0 particle/accuracy_6_of_7‚1 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.5555555820465088 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 :"Purity/1FRt*SRqqt3FBt/event_purity‚0.02777777798473835 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ,jet/accuracy_3_of_4‚0.025770392268896103 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.23820030689239502 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.15563608705997467 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.11056467890739441 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.1666666716337204 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.15724022686481476 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 particle/accuracy_3_of_3‚1 ,particle/sensitivity‚0.24307286739349365 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.27144908905029297 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.14670683443546295 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9956948161125183 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 :"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.34051743149757385 *jet/accuracy_1_of_2‚0.5933186411857605 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9333333373069763 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9219577312469482 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.2949201464653015 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.08510638028383255 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.007946529425680637 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.34375 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.2619685232639313 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.22602339088916779 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 particle/accuracy_1_of_4‚1 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.4285714328289032 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.09776625782251358 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9935897588729858 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 7validation_average_jet_accuracy‚0.31863293051719666 jet/accuracy_4_of_6‚0 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.095238097012043 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.1428571492433548 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.3076989948749542 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.27587902545928955 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.002889381255954504 5 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.99622642993927 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9956948161125183 :"Puritn”;wy/1FRt0SRqqt*FBt/event_purity‚0.08253729343414307 particle/accuracy_2_of_2‚1 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 epoch‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 +jet/accuracy_2_of_3‚0.11868831515312195 particle/accuracy_1_of_6‚1 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 *"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.5 ("Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9897347092628479 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 ("Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9852989315986633 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 ;"Purity/1FRt*SRqqt2FBt/event_purity‚0.051590751856565475 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.15325669944286346 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.5075757503509521 7 Purity/2FRt0SRqqt1FBt/FBt_purity‚0.9940476417541504 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.27178969979286194 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 *jet/accuracy_1_of_1‚0.3500213325023651 jet/accuracy_4_of_5‚0 ,"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.125 jet/accuracy_4_of_7‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.10000000149011612 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.22945907711982727 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.3294546902179718 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.2736682593822479 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.18914705514907837 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.2851990759372711 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.07575757056474686 6 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.984627366065979 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 *jet/accuracy_1_of_5‚0.7840117812156677 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.2222222238779068 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.22029633820056915 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 % _timestamp‚1.7885228979748118e+09 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.3296022117137909 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9845783114433289 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.13850151002407074 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.27964451909065247 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.3555610477924347 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 + Purity/2FRt2SRqqt1FBt/FRt_purity‚0.3125 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.2715517282485962 +"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.25 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.15102194249629974 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.1852608323097229 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.1117895171046257 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.3205678164958954 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚0 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.16254638135433197 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9825173020362854 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.16429202258586884 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_1‚1 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 +jet/accuracy_2_of_6‚0.20512820780277252 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.37173545360565186 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 :"Purity/2FRt1SRqqt1FBt/event_purity‚0.01785714365541935 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.17048294842243195 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.17496705055236816 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 & Purity/*FRt2SRqqt3FBt/FBt_purity‚1 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 (particle/accuracy‚0.8222914338111877 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 +"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.25 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9642857313156128 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.2071743607521057 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.002381041878834367 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.08019164949655533 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.3947368562221527 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1268998235464096 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.3379662036895752 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9635416865348816 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 +jet/accuracy_2_of_4‚0.22064168751239777 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.36000001430511475 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.15729494392871857 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.2523239254951477 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9690539836883545 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1356145739555359 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.1067340075969696 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.41027548909187317 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9825683832168579 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.4101613163948059 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.07499999552965164 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.09195402264595032 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.013888888992369175 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 *loss/val_total_loss‚58.641594073572314 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.5926128625869751 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.2966269850730896 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.1662309318780899 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.2083333283662796 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.23806782066822052 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1041666716337204 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 ("Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.2970939874649048 particle/accuracy_5_of_7‚1 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 +particle/specificity‚0.9294711351394653 jet/accuracy_1_of_7‚0 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.20277778804302216 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.18518517911434174 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.08380171656608582 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9770114421844482 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9951940178871155 ("Purity/2FRt*SRqqt0FBt/event_purity‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 jet/accuracy_5_of_7‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.21441911160945892 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2848984897136688 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 +jet/accuracy_2_of_5‚0.24328595399856567 'particle/f_score‚0.2992917597293854 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.1180555522441864 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.16339637339115143 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_7‚1 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ("Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.26742446422576904 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.5227272510528564 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.0404294990003109 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9225146770477295 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 jet/accuracy_7_of_7‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9878804683685303 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9594109058380127 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.7053794860839844 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.992639422416687 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0022675737272948027 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.12501008808612823 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.2619539797306061 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9849578142166138 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_4_of_6‚1 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9956778287887573 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9920635223388672 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.0833333358168602 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9835266470909119 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.21120689809322357 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.550000011920929 6 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.986067533493042 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 -jet/accuracy_4_of_4‚0.0030423279386013746 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_5_of_6‚1 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 *"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.5 _step‚98 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.1854531317949295 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9877192974090576 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ("Purity/2FRt0SRqqt0FBt/event_purity‚0 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.24828870594501495 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 trainer/global_step‚1208 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.2748599052429199 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.13879141211509705 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.13425925374031067 :"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.35879629850387573 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9523810148239136 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9941313862800598 <"Purity/2FRt*SRqqt*FBt/event_purity‚0.0018479771679267287 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9964672327041626 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.15740744769573212 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.2142857164144516 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.4375 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9789462685585022 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.29763051867485046 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.0708044171333313 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.1723482757806778 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.08596938848495483 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 8"Purity/*FRt0SRqqt3FBt/event_purity‚0.095238097012043 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2743055522441864 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.30223211646080017 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.2083333283662796 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.3222093880176544 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 :"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.35087719559669495 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6182194352149963 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.24642224609851837 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.005916070193052292 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 particle/accuracy_4_of_7‚1 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1614340841770172 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9778561592102051 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9633333683013916 jet/accuracy_3_of_7‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.2704177796840668 particle/accuracy_6_of_6‚1 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.0833333358168602 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.3333333432674408 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.16243843734264374 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.3438422679901123 ,jet/accuracy_2_of_2‚0.047107502818107605 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 & Purity/4FRt1SRqqt*FBt/FRt_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.08593720942735672 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.3529411852359772 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.1287277489900589 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 & Purity/4FRt*SRqqt*FBt/FRt_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,jet/accuracy_3_of_3‚0.009137182496488094 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9184027910232544 jet/accuracy_6_of_7‚0 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.17731748521327972 ;"Purity/2FRt0SRqqt1FBt/event_purity‚0.011904762126505375 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.07726865261793137 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 particle/accuracy_1_of_5‚1 jet/accuracy_6_of_6‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9444444179534912 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.35823625326156616 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.22245068848133087 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.002917833859100938 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 +"Purity/0FRt*SRqqt4FBt/event_purity‚0.25 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚0 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.23806782066822052 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.3611111044883728 particle/accuracy_2_of_7‚1 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.9166666865348816 _runtime‚ 2136.19894201 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.03125 + Purity/1FRt0SRqqt3FBt/FRt_purity‚0.3125 :"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.38411223888397217 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 jet/accuracy_5_of_6‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.0985746756196022 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.10000000149011612 *jet/accuracy_1_of_6‚0.8461538553237915 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 & Purity/0FRt2SRqqt3FBt/FBt_purity‚1 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.021551724523305893 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 ;"Purity/1FRt*SRqqt*FBt/event_purity‚0.039341311901807785 6 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.971230149269104 6 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.998236358165741 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.31690213084220886 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.15636730194091797 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.24698275327682495 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.04764464870095253 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.2142857164144516 particle/accuracy_3_of_7‚1 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.24712643027305603 particle/accuracy_3_of_5‚1 particle/accuracy_3_of_6‚1 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.2638888955116272 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_6‚1 jet/accuracy_2_of_7‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9843390583992004 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.19900479912757874 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9911186695098877 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.7662835717201233 particle/accuracy_4_of_5‚1 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.2750000059604645 particle/accuracy_7_of_7‚0.5 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.13123124837875366 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.10000000149011612 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.2772643566131592 7"Purity/0FRt0SRqqt*FBt/event_purity‚0.80561763048172 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 particle/accuracy_2_of_4‚1 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 +"Purity/0FRt0SRqqt4FBt/event_purity‚0.25 ,jet/accuracy_3_of_6‚0.057692307978868484 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9700201153755188 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_1_of_2‚1 ,"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.125 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.25531309843063354 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9904622435569763 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.2676083743572235 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3520282208919525 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.03800273314118385 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.1631327122449875 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9255319237709045 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9645384550094604 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.24122807383537292 :"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.30560699105262756 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.11873605847358704 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.18682323396205902 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.4309028089046478 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2291666567325592 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.009050179272890091 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2447042614221573 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 & Purity/2FRt*SRqqt3FBt/FRt_purity‚1 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.31673479080200195 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.03940335288643837 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.08438822627067566 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.24534553289413452 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 particle/accuracy_4_of_4‚1 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.980311393737793 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.16997353732585907 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.24669522047042847 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.2348484992980957 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.2361111044883728 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.13426841795444489 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1041666716337204 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.005023447331041098 particle/accuracy_3_of_4‚1 :"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.28026556968688965 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.0023547881282866 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.109375 + Purity/1FRt3SRqqt1FBt/FRt_purity‚0.3125 ,"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.375 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.8333333134651184 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.006569664925336838 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.07027027010917664 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.08024691790342331 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.1666666716337204 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.07407407462596893b,1äX‘º‚(:† šãêÔÙç„ú gpu.0.gpu‚19.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚104.298' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.8596 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ýCóÎh»‚(:Ý šãêÔ¦ªöú* proc.memory.percent‚17.804574966430664 network.sent‚ 2.054893e+06 network.recv‚718903% disk./.usageGB‚1176.0122032165527 disk.dm-1.out‚61.59375 disk.sda.out‚61.59375$ memory_percent‚97.85444736480713 cpu‚25.353176202376083 proc.cpu.threads‚17# disk.nvme0n1.out‚39884.99609375 disk.dm-1.in‚0 disk.dm-0.in‚ 4852.76171875 disk.sda3.in‚0( proc.memory.availableMB‚ 351.52734375" proc.memory.rssMB‚ 2917.1015625# disk.nvme0c0n1.in‚ 4852.76171875 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda.in‚0 disk.sda3.out‚61.59375) disk./.usagePercent‚78.92316569782493% disk.nvme0c0n1.out‚39884.99609375! disk.nvme0n1.in‚ 4852.76171875 disk.drbd1260.out‚ 0.29296875 disk.dm-0.out‚39884.99609375”Ǩ‘¼‚(:† ©ãêÔ¦‰Áö gpu.0.gpu‚26.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚100.548' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.1096 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹1pν‚(:è ©ãêÔêºÜ÷ disk.sda.in‚0& disk.nvme0n1.out‚40421.53369140625 disk.dm-0.in‚ 4852.76171875 disk.sda3.in‚0# disk.nvme0c0n1.in‚ 4852.76171875( disk.nvme0c0n1.out‚40421.53369140625# memory_percent‚97.8562593460083 cpu‚25.35289197536283% disk./.usageGB‚1176.0122032165527 disk.sda1.in‚0 disk.drbd1260.in‚0 disk.sda3.out‚ 61.9140625* proc.memory.percent‚17.804574966430664 network.recv‚724937! disk.nvme0n1.in‚ 4852.76171875 disk.sda1.out‚0# disk.dm-0.out‚ë[¤Ë[40421.53369140625 disk.sda.out‚ 61.9140625( proc.memory.availableMB‚ 351.23046875" proc.memory.rssMB‚ 2917.1015625 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda2.out‚0 network.sent‚ 2.178536e+06) disk./.usagePercent‚78.92316569782493 disk.dm-1.in‚0 disk.dm-1.out‚ 61.9140625 disk.drbd1260.out‚0.296875±Š¼‘¾‚(:† ¸ãêÔ–°±ú gpu.0.gpu‚31.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚110.013' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.0026 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¢ž>p¿‚(:å ¸ãêÔý¯†û network.sent‚ 2.184707e+06 network.recv‚729504! disk.nvme0n1.in‚ 4977.76953125& disk.nvme0n1.out‚40739.46337890625 disk.dm-1.in‚0 disk.sda2.out‚0 disk.sda.in‚0# disk.nvme0c0n1.in‚ 4977.76953125( disk./.usagePercent‚78.9164580444203 disk.sda1.out‚0 disk.dm-1.out‚ 62.359375 disk.drbd1260.in‚0 disk.sda.out‚ 62.359375 disk.sda3.out‚ 62.359375$ memory_percent‚97.85647392272949' proc.memory.availableMB‚ 351.1953125 disk.sda1.in‚0 disk.drbd1260.out‚0.296875 disk.sda3.in‚0# proc.memory.rssMB‚ 2917.10546875* proc.memory.percent‚17.804598808288574 cpu‚25.352233264631508 proc.cpu.threads‚17$ disk./.usageGB‚1175.912254333496 disk.sda2.in‚0 disk.dm-0.in‚ 4977.76953125# disk.dm-0.out‚40739.46337890625( disk.nvme0c0n1.out‚40739.46337890625Œš 1sÀ‚(:è ÇãêÔø disk.drbd1260.in‚0" disk.dm-0.out‚41079.5498046875 disk.sda3.in‚0 disk.sda3.out‚ 63.0078125# disk.nvme0c0n1.in‚ 4977.76953125! disk.nvme0n1.in‚ 4977.76953125 disk.drbd1260.out‚ 0.30078125 disk.sda2.out‚0 disk.sda.in‚0 disk.sda.out‚ 63.0078125' disk.nvme0c0n1.out‚41079.5498046875% disk.nvme0n1.out‚41079.5498046875 disk.sda1.in‚0$ memory_percent‚97.85802364349365 network.sent‚ 2.189911e+06% disk./.usageGB‚1175.9908561706543 disk.dm-1.out‚ 63.0078125 disk.dm-0.in‚ 4977.76953125 cpu‚25.350383164716842 disk.dm-1.in‚0 disk.sda2.in‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 350.94140625! proc.memory.rssMB‚ 2917.109375* proc.memory.percent‚17.804622650146484 proc.cpu.threads‚17 network.recv‚733325) disk./.usagePercent‚78.92173307966331"7å_‘Á‚(:† ÇãêÔšç°ù gpu.0.gpu‚27.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚108.712' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7424 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0!L]Y2Â- _wandbruntime‚2200  _runtime‚2200@á ÆÃÀ % _timestamp‚1.7885229624070427e+09 _runtime‚2200.627860429 _step‚99 & lr-AdamW/pg1‚0.0009188504630582964 & lr-AdamW/pg2‚0.0009188504630582964 trainer/global_step‚1249ƒ&1ÊÄÄ trainer/global_step‚1249 % _timestamp‚1.7885229624070427e+09 _runtime‚2200.627860429 _step‚99 & lr-AdamW/pg1‚0.0009188504630582964 & lr-AdamW/pg2‚0.0009188504630582964cÚ7C2Å- _wandbruntime‚2200  _runtime‚2200æz0<Æ'ÆÀO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚33999764 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 _step‚100 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9046537 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚23825307 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2484424142336 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚51019435526144 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚56564940903936 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2477375607209 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚33999764 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚687 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2529763359744 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚53503607997952 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚51013940274789 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚53503859668480 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚54034876438016 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚51019417523200 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9047224 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚48536793648656 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚51019417523200 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚51019435526144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14247186 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚54034852497920 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚56564615857664 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2530064465920 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚57825069 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚53503607997952 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23293721 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚53503859668480 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚57851413 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚23825307 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚34026106 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚689 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2484190474752 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14247184 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 % _timestamp‚1.7885229624092858e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚23825305 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2484424142336 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23825305 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2484190474752 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2477146626133_‘ö¨Ê'ÇÄO D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2477375607209 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚33999764 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚687 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2529763359744 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚53503607997952 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚51013940274789 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚53503859668480 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚54034876438016 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚51019417523200 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9047224 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚48536793648656 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚51019417523200 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚51019435526144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14247186 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚54034852497920 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚56564615857664 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2530064465920 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚57825069 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚53503607997952 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23293721 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚53503859668480 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚57851413 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚23825307 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚34026106 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚689 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2484190474752 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14247184 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 % _timestamp‚1.7885229624092858e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚23825305 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2484424142336 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23825305 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2484190474752 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2477146626133 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚33999764 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 _step‚100 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9046537 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚23825307 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2484424142336 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚51019435526144 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚56564940903936døn»2È- _wandbruntime‚2202  _runtime‚2202ZUº4"ÉÊM REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚34052824 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2486388181504 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2486154500608 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚57896406 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚51053675141843 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚23843582 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚53545109744640 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚56608609729024 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚56608295181824 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚54076560408064 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚53545109744640 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23311694 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚48574324006080 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚51058721563136 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚51058703560192 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚57896406 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚23843580 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚34052824 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚57870036 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚51058721563136 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚51058703560192 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314547200 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2479334096403 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14257986 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚51053675141843 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚23843582 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚53545109744640 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚56608609729024 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚56608295181824 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚54076560408064 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚53545109744640 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23311694 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚48574324006080 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚51058721563136 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚51058703560192 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚57896406 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚23843580 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚34052824 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚57870036 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚51058721563136 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚51058703560192 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314547200 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2479334096403 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14257986 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚55566125554176 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2575313647717 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 trainer/global_step‚1299 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚24739057 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24180851 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2582406181376 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚52983719372800 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2629550539264 $ _timestamp‚1.788523053023927e+09 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚52988396186497 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚52983719372800 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚35364058 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚56119074940416 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2629834868224 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚701 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚56119051000320 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚60076771 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9394266 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚58748909808640 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308269056 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24181552 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2582406181376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚60103115 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284328960 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 _runtime‚2291.244985572 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35337716 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚24739057 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚699 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9393567 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚52988642206933 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚55566125554176 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚24739055ü̲±ßÃO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚24739057 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24180851 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2582406181376 [HDeviceStatsMonitor.on_train_batch_©bú6&start/allocated_bytes.large_pool.freed‚52983719372800 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2629550539264 $ _timestamp‚1.788523053023927e+09 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚52988396186497 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚52983719372800 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚35364058 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚56119074940416 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2629834868224 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚701 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚56119051000320 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚60076771 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9394266 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚58748909808640 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308269056 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24181552 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2582406181376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚60103115 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284328960 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 _runtime‚2291.244985572 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35337716 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚24739057 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚699 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9393567 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚52988642206933 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚55566125554176 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚24739055 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚50413311519856 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚52983737375744 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14787284 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚24739055 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚55566377224704 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2582639848960 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚55566125554176 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2575313647717 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 trainer/global_step‚1299hüŒ¸92à- _wandbruntime‚2292  _runtime‚2292P]{Ð&áÊM =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚53023005409792 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚58792592944128 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚58792282591232 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚53023023412736 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚35390776 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 _step‚105 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _runtime‚2292.869172082 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14798088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚60148108 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚60121738 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚24757332 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚35390776 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2584370207232 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚55607627300864 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2631547620864 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚60148108 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚35364408 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 % _timestamp‚1.7885230546485546e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2577043155627 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2577272136911 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚53023023412736 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2584603888128 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286412800 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2584370207232 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚24757330 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚674 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24757330 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚55607375617024 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚50450841877280 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24198682 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14798086 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚50450858916640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2631834033664 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 trainer/global_step‚1299 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284›ób¢Ô&âÎM M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚58792592944128 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚58792282591232 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚53023023412736 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚35390776 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 _step‚105 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _runtime‚2292.869172082 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14798088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚60148108 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚60121738 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚24757332 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚35390776 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2584370207232 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚55607627300864 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2631547620864 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚60148108 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚35364408 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 % _timestamp‚1.7885230546485546e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2577043155627 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2577272136911 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚53023023412736 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2584603888128 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286412800 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2584370207232 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚24757330 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚674 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24757330 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚55607375617024 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚50450841877280 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24198682 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14798086 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚50450858916640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2631834033664 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 trainer/global_step‚1299 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚53023005409792 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309iˆû!s2ã- _wandbruntime‚2292  _runtime‚2292GE»XäÒ 1loss/SRqqt3/detection_loss‚0.9847149412005635 %loss/total_loss‚57.89783888541885 0loss/FBt3/assignment_loss‚0.8311347663402557 .loss/FRt4/detection_loss‚1.988489396042294 trainer/global_step‚1299 /loss/FRt1/detection_loss‚2.8796339360150425 /loss/FRt2/detection_loss‚3.2013443261384964 0loss/FRt1/assignment_loss‚6.0312443213029345 /loss/FRt3/detection_loss‚2.0175252721186667 % _timestamp‚1.7885230546522017e+09 _runtime‚2292.871653361 0loss/FRt4/assignment_loss‚3.2803683797518413 1loss/SRqqt1/assignment_loss‚2.648273097972075 1loss/SRqqt3/assignment_loss‚2.045561760663986 0loss/FBt2/assignment_loss‚0.8026707569758097 0loss/SRqqt1/detection_loss‚2.938115273912748 epoch‚1 2loss/SRqqt2/assignment_loss‚2.8712822003290057 1loss/FBt4/assignment_loss‚0.21629328121880353 1loss/SRqqt4/detection_loss‚3.0197732044117793 1loss/SRqqt4/assignment_loss‚2.235956913658551 0loss/FBt1/assignment_loss‚0.7485560725100034 /loss/FBt1/detection_loss‚2.8319598784049353 /loss/FBt3/detection_loss‚2.1070965826511383 _step‚106 1loss/SRqqt2/detection_loss‚2.6881739515811205 .loss/FBt2/detection_loss‚2.566337908307711 /loss/FBt4/detection_loss‚0.5964728716672176 /loss/FRt2/assignment_loss‚5.657430599133174 /loss/FRt3/assignment_loss‚2.709429193110693€±ÚÕ\åÖ /loss/FRt1/detection_loss‚2.8796339360150425 /loss/FRt2/detection_loss‚3.2013443261384964 0loss/FRt1/assignment_loss‚6.0312443213029345 /loss/FRt3/detection_loss‚2.0175252721186667 % _timestamp‚1.7885230546522017e+09 _runtime‚2292.871653361 0loss/FRt4/assignment_loss‚3.2803683797518413 1loss/SRqqt1/assignment_loss‚2.648273097972075 1loss/SRqqt3/assignment_loss‚2.045561760663986 0loss/FBt2/assignment_loss‚0.8026707569758097 0loss/SRqqt1/detection_loss‚2.938115273912748 epoch‚1 2loss/SRqqt2/assignment_loss‚2.8712822003290057 1loss/FBt4/assignment_loss‚0.21629328121880353 1loss/SRqqt4/detection_loss‚3.0197732044117793 1loss/SRqqt4/assignment_loss‚2.235956913658551 0loss/FBt1/assignment_loss‚0.7485560725100034 /loss/FBt1/detection_loss‚2.8319598784049353 /loss/FBt3/detection_loss‚2.1070965826511383 _step‚106 1loss/SRqqt2/detection_loss‚2.6881739515811205 .loss/FBt2/detection_loss‚2.566337908307711 /loss/FBt4/detection_loss‚0.5964728716672176 /loss/FRt2/assignment_loss‚5.657430599133174 /loss/FRt3/assignment_loss‚2.709429193110693 1loss/SRqqt3/detection_loss‚0.9847149412005635 %loss/total_loss‚57.89783888541885 0loss/FBt3/assignment_loss‚0.8311347663402557 .loss/FRt4/detection_loss‚1.988489396042294 trainer/global_step‚1299j鯊¸ôæ‚(:† °äêÔÆ‘®÷ gpu.0.gpu‚14.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚130.243' gpu.0.enforcedPowerLimitWatts‚500.ܲÃ0 gpu.0.powerPercent‚26.0486 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ÞEYsç‚(:è °äêԜשú proc.cpu.threads‚17) disk./.usagePercent‚78.93578789329311 disk.sda1.in‚0 disk.dm-1.out‚ 65.6015625# disk.dm-0.out‚42963.00732421875 disk.sda.in‚0 disk.sda.out‚ 65.6015625! proc.memory.rssMB‚ 2917.109375 cpu‚25.34659313631178 disk.sda3.in‚0( disk.nvme0c0n1.out‚42963.00732421875 disk.dm-1.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 65.6015625# disk.nvme0c0n1.in‚ 5283.73046875! disk.nvme0n1.in‚ 5283.73046875 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.sda2.in‚0$ memory_percent‚99.93808269500732' proc.memory.availableMB‚ 10.14453125 network.sent‚ 2.344543e+06 network.recv‚764268$ disk./.usageGB‚1176.200283050537& disk.nvme0n1.out‚42963.00732421875 disk.drbd1260.out‚ 0.31640625 disk.dm-0.in‚ 5283.73046875* proc.memory.percent‚17.804622650146484……Örè‚(:ç ¿äêÔíÝ…ù disk.sda2.out‚0 cpu‚25.346220491445028 disk.sda.out‚ 66.27734375 disk.sda3.out‚ 66.27734375 disk.drbd1260.out‚ 0.3203125 disk.dm-0.in‚ 5298.56640625$ memory_percent‚99.93987083435059 proc.cpu.threads‚17 network.recv‚769631 disk.sda.in‚0! disk.nvme0n1.in‚ 5298.56640625 disk.dm-1.in‚0 disk.dm-1.out‚ 66.27734375 disk.sda2.in‚0" disk.dm-0.out‚43260.4013671875% proc.memory.availableMB‚ 9.8515625 network.sent‚ 2.438894e+06% disk./.usageGB‚1176.3252792358398 disk.sda3.in‚0% disk.nvme0n1.out‚43260.4013671875 disk.sda1.in‚0 disk.sda1.out‚0! proc.memory.rssMB‚ 2917.109375* proc.memory.percent‚17.804622650146484) disk./.usagePercent‚78.94417649216756# disk.nvme0c0n1.in‚ 5298.56640625' disk.nvme0c0n1.out‚43260.4013671875 disk.drbd1260.in‚0(q:õœé‚(:‘ ¿äêÔèúù gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚140.227' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.045399999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0üœ_j‘ê‚(:† ÎäêÔ”–èõ gpu.0.gpu‚44.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚127.894' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.5788 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0vÛ¥Âtë‚(:é ÎäêÔͱ×÷' disk.nvme0c0n1.out‚43601.5810546875) disk./.usagePercent‚78.94417034798674% disk./.usageGB‚1176.3251876831055 disk.sda1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.3203125! proc.memory.rssMB‚ 2917.109375* proc.memory.percent‚17.804622650146484" disk.dm-0.out‚43601.5810546875% disk.nvme0n1.out‚43601.5810546875 cpu‚25.346063911673774 proc.cpu.threads‚17 disk.sda3.out‚ 66.77734375 network.sent‚ 2.443981e+06 network.recv‚773116 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 5298.56640625 disk.sda.out‚ 66.77734375! disk.nvme0n1.in‚ 5298.56640625 disk.sda.in‚0 disk.sda3.in‚0# disk.nvme0c0n1.in‚ 5298.56640625 disk.sda1.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 66.77734375$ memory_percent‚99.93836879730225' proc.memory.availableMB‚ 10.09765625‰$oX‘ì‚(:† ÝäêÔºÄú gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚114.461' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.8922 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0VðOpí‚(:å ÝäêÔÒŒ¨û network.sent‚ 2.449878e+06) disk./.usagePercent‚78.95937309940007 disk.sda.in‚0# disk.nvme0c0n1.in‚ 5298.56640625 disk.drbd1260.in‚0* proc.memory.percent‚17.804622650146484 cpu‚25.345381091942905 network.recv‚777461 disk.sda2.in‚0( disk.nvme0c0n1.out‚43996.50830078125& disk.nvme0n1.out‚43996.50830078125! proc.memory.rssMB‚ 2917.109375 proc.cpu.threads‚17% disk./.usageGB‚1176.5517196655273# disk.dm-0.out‚43996.50830078125 disk.sda3.in‚0 disk.sda3.out‚ 67.203125 disk.dm-1.in‚0 disk.dm-1.out‚ 67.203125% proc.memory.availableMB‚ 10.109375 disk.sda2.out‚0 disk.dm-0.in‚ 5298.56640625 disk.sda.out‚ 67.203125! disk.nvme0n1.in‚ 5298.56640625 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.out‚ 0.32421875$ memory_percent‚99.93829727172852˜‰zrî‚(:„ ìäêÔÖÙãö gpu.0.gpu‚8.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚96.273' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.2546 gpu.0.smClock‚1305 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿'¹qï‚(:æ ìäêÔŒùÊù% disk./.usageGB‚1176.5753479003906" disk.dm-0.out‚44486.2666015625' disk.nvme0c0n1.out‚44486.2666015625$ memory_percent‚99.93839263916016$ proc.memory.availableMB‚10.09375! proc.memory.rssMB‚ 2917.109375 proc.cpu.threads‚17 network.sent‚ 2.45589e+06 network.recv‚781555) disk./.usagePercent‚78.96095881006715 disk.sda2.out‚0 disk.dm-0.in‚ 5298.56640625 disk.sda.out‚ 67.57421875 disk.sda3.in‚0 disk.sda3.out‚ 67.57421875 disk.sda2.in‚0 disk.sda.in‚0% disk.nvme0n1.out‚44486.2666015625 disk.sda1.in‚0 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.32421875* proc.memory.percent‚17.804622650146484# disk.nvme0c0n1.in‚ 5298.56640625! disk.nvme0n1.in‚ 5298.56640625 disk.sda1.out‚0 disk.dm-1.out‚ 67.57421875 cpu‚25.345136308713542#T&‘ð‚(:† ûäêÔè—ûõ gpu.0.gpu‚30.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚113.954' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.7908 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0» @ sñ‚(:è ûäêÔÀúæ÷' disk.nvme0c0n1.out‚44544.9443359375! disk.nvme0n1.in‚ 5415.48828125 disk.sda1.out‚0 disk.dm-1.out‚ 67.98828125 disk.drbd1260.in‚0) disk./.usagePercent‚78.96096162615002% disk./.usageGB‚1176.5753898620605 disk.sda1.in‚0 disk.dm-1.in‚0 disk.dm-0.in‚ 5415.48828125" disk.dm-0.out‚44544.9443359375 disk.sda3.in‚0 disk.sda3.out‚ 67.98828125 network.sent‚ 2.46025e+06 disk.sda2.out‚0 disk.sda.in‚0$ memory_percent‚99.93689060211182! proc.memory.rssMB‚ 2917.109375* proc.memory.percent‚17.804622650146484 proc.cpu.threads‚17% disk.nvme0n1.out‚44544.9443359375 disk.sda2.in‚0 disk.sda.out‚ 67.98828125' proc.memory.availableMB‚ 10.33984375 cpu‚25.34468894107871 disk.drbd1260.out‚ 0.33203125 network.recv‚784764# disk.nvme0c0n1.in‚ 5415.48828125δíô2ò- _wandbruntime‚2382  _runtime‚2382Ï‚H¡ÇóÁ & lr-AdamW/pg1‚0.0009185058196327014 & lr-AdamW/pg2‚0.0009185058196327014 trainer/global_step‚1349 % _timestamp‚1.7885231439403417e+09 _runtime‚2382.160471769 _step‚107ݾ£ËôÅ trainer/global_step‚1349 % _timestamp‚1.7885231439403417e+09 _runtime‚2382.160471769 _step‚107 & lr-AdamW/pg1‚0.0009185058196327014 & lr-AdamW/pg2‚0.0009185058196327014k5u‡î2õ- _wandbruntime‚2382  _runtime‚2382ÔÕ@Æ'öÀO QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚58203273442816 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚54948021222400 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 306171904 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚60932634843136 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚54948039225344 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚57628643110400 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚62328473 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚54948039225344 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚62354817 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚57628894780928 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚25652807 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9739354 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 _runtime‚2382.163986515 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚36675668 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 282231808 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25652805 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚36702010 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚689 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚25652805 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚57628894780928 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2680855555584 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 trainer/global_step‚1349 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2680621888000 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚57628643110400 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚36675668 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 282231808 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25652805 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚36702010 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚689 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚25652805 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚57628894780928 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2680855555584 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 trainer/global_step‚1349 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2680621888000 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚57628643110400 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚57628643110400 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚62328473lW¹‹ý2ø- _wandbruntime‚2384  _runtime‚2384ZÔ$Ÿ”ùÉM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚54987325262336 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9747047 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚52327376787840 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25085235 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 312450048 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚62399810 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚62399810 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚36728728 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25671082 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25084565 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15338186 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚55002340944615 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚36728728 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2682819594752 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚52327359748480 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2682585913856 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2731381271040 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚54987307259392 YGDeviceStatsMonitor.on_train_batch_end/activ52¨;e_bytes.small_pool.allocated‚ 2682819594752 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚25671082 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚668 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚54987325262336 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 _runtime‚ 2384.52256898 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚62373440 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚58244957412864 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2675210177419 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚25671080 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚57670144857088 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2682585913856 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚36702360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚36702360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚670 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚57669893173248 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 % _timestamp‚1.7885231462930377e+09äò)hÓ&úÍM QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9746379 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚54987307259392 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15338188 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚55002340944615 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚36728728 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2682819594752 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚52327359748480 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2682585913856 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2731381271040 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚54987307259392 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2682819594752 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚25671082 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚668 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚54987325262336 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 _runtime‚ 2384.52256898 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚62373440 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚58244957412864 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2675210177419 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚25671080 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚57670144857088 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2682585913856 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚36702360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚36702360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚670 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚57669893173248 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 % _timestamp‚1.7885231462930377e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚54987325262336 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9747047 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚52327376787840 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25085235 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 312450048 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚62399810 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚62399810 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚36728728 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25671082 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25084565 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15338186 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 O)†‚(:… ÆåêÔèÒ÷ gpu.0.gpu‚22.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚99.611' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.9222 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚03­¢q‡‚(:æ ÆåêÔ·‚£ù disk.sda1.out‚0 disk.dm-1.out‚ 70.265625 disk.drbd1260.in‚0 disk.sda.out‚ 70.265625 disk.sda3.in‚0$ memory_percent‚99.93727207183838 cpu‚25.34301450749737) disk./.usagePercent‚78.99451294955743! disk.nvme0n1.in‚ 5560.95703125& disk.nvme0n1.out‚46126.04931640625 disk.sda1.in‚0 disk.dm-1.in‚0 disk.sda2.in‚0 disk.sda.in‚0* proc.memory.percent‚17.804622650146484 network.sent‚ 2.590491e+06% disk./.usageGB‚1177.0753288269043# disk.nvme0c0n1.in‚ 5560.95703125 disk.dm-0.in‚ 5560.95703125 disk.sda3.out‚ 70.265625 proc.cpu.threads‚17( disk.nvme0c0n1.out‚46126.04931640625 disk.drbd1260.out‚ 0.33984375 disk.sda2.out‚0# disk.dm-0.out‚46126.04931640625' proc.memory.availableMB‚ 10.27734375! proc.memory.rssMB‚ 2917.109375 network.recv‚807450½gˆ‚(:„ ÕåêÔ³‹¤÷ gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.031' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.6062 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0†‡è¸V‰‚(:Ë ÕåêÔ÷ªùù network.sent‚ 2.596371e+06 network.recv‚812083 disk.dm-1.in‚0 disk.sda2.in‚0 disk.dm-0.in‚ 5560.95703125 cpu‚25.34264404998156) disk./.usagePercent‚78.99293952725198 disk.dm-1.out‚70.5625 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda3.out‚70.5625! disk.nvme0n1.in‚ 5560.95703125 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.out‚0.34375 disk.sda2.out‚0 disk.dm-0.out‚ 46356.78125$ memory_percent‚99.93879795074463! proc.memory.rssMB‚ 2917.109375* proc.memory.percent‚17.804622650146484 proc.cpu.threads‚17% disk./.usageGB‚1177.0518836975098 disk.sda.out‚70.5625 disk.sda3.in‚0# disk.nvme0c0n1.in‚ 5560.95703125" disk.nvme0c0n1.out‚ 46356.78125 disk.nvme0n1.out‚ 46356.78125' proc.memory.availableMB‚ 10.02734375Ù—}¾2Š- _wandbruntime‚2472  _runtime‚2472–—/lÇ‹Á & lr-AdamW/pg1‚0.0009181899722065781 & lr-AdamW/pg2‚0.0009181899722065781 trainer/global_step‚1399 % _timestamp‚1.7885232342185564e+09 _runtime‚2472.439186317 _step‚111Ë+­{ˌŠ& lr-AdamW/pg1‚0.0009181899722065781 & lr-AdamW/pg2‚0.0009181899722065781 trainer/global_step‚1399 % _timestamp‚1.7885232342185564e+09 _runtime‚2472.439186317 _step‚111o"V¤2- _wandbruntime‚2472  _runtime‚2472W«ŸŽÂO KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚54166364301616 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚56912323072000 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚26566555 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2829604770304 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚64606519 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10086734 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚38013620 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 304074752 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚59691160666624 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2779071262208 trainer/global_step‚1399 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15867486 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2778837594624 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚63116772640768 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 280134656 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25954885 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚665 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2829324635648 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚56937308009913 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚38013620 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚59691412337152 _runtime‚2472.443281708 Q•Ì©)_step‚112 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10087399 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚59691412337152 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚59691160666624 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚56912341074944 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚26566555 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚56912323072000 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚667 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚60287448005120 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2771189728733 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 % _timestamp‚1.7885232342223482e+09 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚56912341074944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0ºè[Ì'ÆO VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 280134656 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25954885 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚665 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2829324635648 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚56937308009913 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚38013620 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚59691412337152 _runtime‚2472.443281708 _step‚112 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10087399 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚59691412337152 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚59691160666624 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚56912341074944 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚26566555 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚56912323072000 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚667 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚60287448005120 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2771189728733 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 % _timestamp‚1.7885232342223482e+09 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚56912341074944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚54166364301616 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚56912323072000 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚26566555 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2829604770304 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚64606519 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10086734 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚38013620 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 304074752 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚59691160666624 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2779071262208 trainer/global_step‚1399 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15867486 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2778837594624 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚63116772640768p=F¤/2- _wandbruntime‚2474  _runtime‚2474î×Ý<Ò&‘ÌM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15878286 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚56976796856323 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚54203894659040 _runtime‚2474.709709303 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25971704 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26584832 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚38040312 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2781035301376 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚56951609108992 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2772919236643 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310352896 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚59732662413312 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚56951627111936 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚689 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚59732410729472 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚26584830 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25972395 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚38066680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10094107 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2773148217927 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚63160764241920 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2831608326656 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚26584830 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚64651512 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286412800 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚64651512 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚64625142 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2780801620480 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚38066680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2831321913856 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2780801620480 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚26584832 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚54203877619680 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 % _timestamp‚1.7885232364826217e+09 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚691 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 _step‚113 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚56977042876967 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582G³ãä ’ÐM VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10094107 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2773148217927 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚63160764241920 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2831608326656 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚26584830 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚64651512 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286412800 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚64651512 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚64625142 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2780801620480 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚38066680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2831321913856 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2780801620480 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚26584832 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚54203877619680 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 % _timestamp‚1.7885232364826217e+09 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚691 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 _step‚113 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚56977042876967 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15878286 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚56976796856323 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚54203894659040 _runtime‚2474.709709303 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25971704 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26584832 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚38040312 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2781035301376 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚56951609108992 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2772919236643 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310352896 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚59732662413312 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚56951627111936 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚689 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚59732410729472 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚26584830 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25972395 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚38066680qI1=e2“- _wandbruntime‚2474  _runtime‚2474àö‚E”¿ /loss/FRt3/assignment_loss‚2.361583261720596 0loss/FBt1/assignment_loss‚0.5283817891902478 0loss/SRqqt4/detection_loss‚2.330665874481201 .loss/FBt1/detection_loss‚2.457871338900398 &loss/total_loss‚52.627956491856544 epoch‚1 2loss/SRqqt3/assignment_loss‚1.2712328576984313 1loss/SRqqt2/assignment_loss‚2.599796701045263 2loss/SRqqt4/assignment_loss‚2.3895355865359305 0loss/FBt2/assignment_loss‚0.5636060318113476 1loss/SRqqt2/detection_loss‚2.7209804966336204 /loss/FBt3/detection_loss‚1.7145862579345703 trainer/global_step‚1399 % _timestamp‚1.7885232364933865e+09 2loss/SRqqt1/assignment_loss‚1.9620791533414055 /loss/FRt1/detection_loss‚2.8972536094727053 /loss/FRt4/detection_loss‚2.1954959478320144 0loss/SRqqt1/detection_loss‚3.216813483658959 /loss/FBt4/detection_loss‚0.7180332440072799 /loss/FRt2/assignment_loss‚5.116422211682355 /loss/FRt2/detection_loss‚3.0003469740902937 _runtime‚2474.713987698 /loss/FRt1/assignment_loss‚5.140017048005135 0loss/FRt4/assignment_loss‚3.6242124016691997 1loss/SRqqt3/detection_loss‚0.9316560027691034 _step‚114 loss/FBt3/assignment_loss‚0 0loss/FBt4/assignment_loss‚0.3375613219824952 /loss/FRt3/detection_loss‚1.7893851255216906 ,loss/FBt2/detection_loss‚2.7604397718723AgäËI•à /loss/FRt2/assignment_loss‚5.116422211682355 /loss/FRt2/detection_loss‚3.0003469740902937 _runtime‚2474.713987698 /loss/FRt1/assignment_loss‚5.140017048005135 0loss/FRt4/assignment_loss‚3.6242124016691997 1loss/SRqqt3/detection_loss‚0.9316560027691034 _step‚114 loss/FBt3/assignment_loss‚0 0loss/FBt4/assignment_loss‚0.3375613219824952 /loss/FRt3/detection_loss‚1.7893851255216906 ,loss/FBt2/detection_loss‚2.7604397718723 /loss/FRt3/assignment_loss‚2.361583261720596 0loss/FBt1/assignment_loss‚0.5283817891902478 0loss/SRqqt4/detection_loss‚2.330665874481201 .loss/FBt1/detection_loss‚2.457871338900398 &loss/total_loss‚52.627956491856544 epoch‚1 2loss/SRqqt3/assignment_loss‚1.2712328576984313 1loss/SRqqt2/assignment_loss‚2.599796701045263 2loss/SRqqt4/assignment_loss‚2.3895355865359305 0loss/FBt2/assignment_loss‚0.5636060318113476 1loss/SRqqt2/detection_loss‚2.7209804966336204 /loss/FBt3/detection_loss‚1.7145862579345703 trainer/global_step‚1399 % _timestamp‚1.7885232364933865e+09 2loss/SRqqt1/assignment_loss‚1.9620791533414055 /loss/FRt1/detection_loss‚2.8972536094727053 /loss/FRt4/detection_loss‚2.1954959478320144 0loss/SRqqt1/detection_loss‚3.216813483658959 /loss/FBt4/detection_loss‚0.7180332440072799rê{–‚(:‚ äåê԰΃ö gpu.0.gpu‚3.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚99.44' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.888 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚08€ëCq—‚(:æ äåêÔ•¿Åø& disk.nvme0n1.out‚46701.23779296875 disk.sda3.in‚0# disk.nvme0c0n1.in‚ 5560.95703125! disk.nvme0n1.in‚ 5560.95703125# memory_percent‚99.9392032623291% proc.memory.availableMB‚ 9.9609375 cpu‚25.34241996410747 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda1.out‚0! proc.memory.rssMB‚ 2917.109375* proc.memory.percent‚17.804622650146484 proc.cpu.threads‚17 network.sent‚ 2.602751e+06% disk./.usageGB‚1177.0666542053223 disk.dm-1.in‚0 disk.drbd1260.out‚0.34375 disk.sda2.in‚0 disk.dm-0.in‚ 5560.95703125# disk.dm-0.out‚46701.23779296875 disk.sda1.in‚0 network.recv‚817091) disk./.usagePercent‚78.99393078842456 disk.dm-1.out‚ 70.99609375 disk.sda2.out‚0 disk.sda.out‚ 70.99609375 disk.sda3.out‚ 70.99609375( disk.nvme0c0n1.out‚46701.23779296875©GçȘ‚(:„ óåêÔдø gpu.0.gpu‚5.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚92.744' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.5488 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Bûft™‚(:é óåêÔÈâŒû! proc.memory.rssMB‚ 2917.109375* proc.memory.percent‚17.804622650146484 proc.cpu.threads‚17 network.sent‚ 2.709918e+06 network.recv‚822348) disk./.usagePercent‚78.99471493950195 disk.sda.in‚0 disk.sda.out‚ 71.3828125 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.dm-1.out‚ 71.3828125' proc.memory.availableMB‚ 10.19140625 cpu‚25.342116828904373 disk.sda3.out‚ 71.3828125# disk.nvme0c0n1.in‚ 5753.62890625% disk./.usageGB‚1177.0783386230469 disk.dm-0.in‚ 5753.62890625# disk.dm-0.out‚47076.45947265625 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.34765625# memory_percent‚99.9377965927124 disk.sda2.in‚0( disk.nvme0c0n1.out‚47076.45947265625! disk.nvme0n1.in‚ 5753.62890625& disk.nvme0n1.out‚47076.45947265625ÂÊoš‚(:„ ‚æêÔÍÞùõ gpu.0.gpu‚20.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.345' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.669 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0⨖Ôo›‚(:ä ‚æêÔÞ•ªø" proc.memory.availableMB‚9.9375 cpu‚25.34182869985675 proc.cpu.threads‚17% disk./.usageGB‚1177.0783386230469 disk.sda.in‚0 disk.sda.out‚ 71.9296875( disk.nvme0c0n1.out‚47417.61181640625 disk.sda1.in‚0 disk.dm-1.out‚ 71.9296875 disk.sda2.in‚0$ memory_percent‚99.93934631347656 network.recv‚825907 disk.sda2.out‚0# disk.dm-0.out‚47417.61181640625! proc.memory.rssMB‚ 2917.109375* proc.memory.percent‚17.804622650146484 network.sent‚ 2.715065e+06) disk./.usagePercent‚78.99471493950195 disk.sda3.in‚0# disk.nvme0c0n1.in‚ 5753.62890625! disk.nvme0n1.in‚ 5753.62890625& disk.nvme0n1.out‚47417.61181640625 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.sda3.out‚ 71.9296875 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.34765625 disk.dm-0.in‚ 5753.62890625¬Y 4œ‚(:… ‘æêÔÙÞå÷ gpu.0.gpu‚6.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚102.421' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.4842 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"Àïip‚(:å ‘æêÔ§ð¶ú network.recv‚830151# disk.nvme0c0n1.in‚ 5753.62890625 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.sda.out‚ 72.234375 disk.sda3.in‚0 disk.sda3.out‚ 72.234375! proc.memory.rssMB‚ 2917.109375 network.sent‚ 2.721057e+06( disk.nvme0c0n1.out‚47668.23876953125& disk.nvme0n1.out‚47668.23876953125 disk.dm-1.out‚ 72.234375 disk.sda2.out‚0 disk.dm-0.in‚ 5753.62890625& proc.memory.availableMB‚ 10.1796875* proc.memory.percent‚17.804622650146484! disk.nvme0n1.in‚ 5753.62890625 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.3515625# disk.dm-0.out‚47668.23876953125$ memory_percent‚99.93786811828613 cpu‚25.341541863856378 proc.cpu.threads‚17) disk./.usagePercent‚78.96552342440265% disk./.usageGB‚1176.6433639526367 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda.in‚0Ç$‚›ž‚(:  æêÔƒ­Ñ÷ gpu.0.gpu‚15.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.564' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚18.712799999999998 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚07±DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚58876624921600 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚58911763921621 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284328960 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚56042865133456 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚61753678222848 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2877286968832 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚39377914 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 _runtime‚2564.958803126 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚61753678222848 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2877286968832 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2877053301248 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚27480307 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚62371646507520 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2869127769241 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10433361 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚39351572 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16407586 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚56042882172816 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚61753929893376 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2929468499968 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2929184171008 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16407584 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚61753929893376 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚66858221 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚682 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚62371670447616 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚65300830678528 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2929468499968 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2929184171008 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16407584 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚61753929893376 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚66858221 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚682 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚62371670447616 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚65300830678528 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚58876624921600 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚58911763921621 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284328960 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚56042865133456 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚61753678222848 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2877286968832 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚39377914 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 _runtime‚2564.958803126 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚61753678222848 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2877286968832 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2877053301248 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚27480307 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚62371646507520 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2869127769241 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 RDDeviceStatsMonitor.on_train_bat¡Øô6Üch_start/active_bytes.small_pool.peak‚ 565024768 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10433361t¥âÖ¨‚(:„ ¾æêÔ“Òòù gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.031' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.6062 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Œ÷v©‚(:ë ¾æêÔÜî“ü disk.sda3.in‚0# disk.nvme0c0n1.in‚ 5907.14453125% disk.nvme0n1.out‚48816.2607421875 disk.sda1.in‚0 disk.dm-1.in‚0 cpu‚25.340505800142427% disk./.usageGB‚1176.6208610534668 disk.sda.in‚0 disk.sda3.out‚ 73.62890625' disk.nvme0c0n1.out‚48816.2607421875 disk.sda1.out‚0 disk.dm-1.out‚ 73.62890625 disk.sda2.in‚0& proc.memory.availableMB‚ 9.49609375 network.recv‚841955 disk.sda.out‚ 73.62890625 disk.drbd1260.in‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 2917.11328125* proc.memory.percent‚17.804646492004395 proc.cpu.threads‚17 network.sent‚ 2.737378e+06) disk./.usagePercent‚78.96401323595818! disk.nvme0n1.in‚ 5907.14453125 disk.drbd1260.out‚ 0.35546875$ memory_percent‚99.94204044342041 disk.dm-0.in‚ 5907.14453125" disk.dm-0.out‚48816.2607421875Ú±¸Ÿ2ª- _wandbruntime‚2567  _runtime‚2567æÒÒ&«ÌM WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66903214 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚65344821429760 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚2567.188504104 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚56080412530240 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 _step‚117 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10440766 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚56080395490880 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚66876844 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚39404632 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 308255744 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2931182696448 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚649 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚39404632 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚58915910958592 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚62413330477568 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚61795179969536 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2879017327104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27498580 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2879017327104 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚65344513174016 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚62413354417664 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚61795179969536 % _timestamp‚1.7885233289670563e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚27498580 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10440119 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2931467012096 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚27498582 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16418388 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26858505 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2871086258435 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66903214 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚65344821429760 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚2567.188504104 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚56080412530240 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 _step‚117 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10440766 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚56080395490880 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚66876844 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚39404632 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 308255744 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2931182696448 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚649 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚39404632 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚58915910958592 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚62413330477568 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚61795179969536 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2879017327104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27498580 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2879017327104 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚65344513174016 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚62413354417664 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚61795179969536 % _timestamp‚1.7885233289670563e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚27498580 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10440119 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2931467012096 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚27498582 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16418388 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26858505 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2871086258435u!pÒ…2­- _wandbruntime‚2567  _runtime‚2567¸=çR®Ì 1loss/SRqqt2/detection_loss‚2.7155534987864285 .loss/FRt2/assignment_loss‚5.39235200881958 0loss/FBt1/assignment_loss‚0.4361360368177636 .loss/FRt2/detection_loss‚3.470706558227539 .loss/FRt3/detection_loss‚1.656569320904581 /loss/FBt1/detection_loss‚2.3704957420175727 /loss/FBt4/detection_loss‚1.2770239258783735 _step‚118 2loss/SRqqt3/assignment_loss‚1.1197596252087718 0loss/FBt4/assignment_loss‚0.3342698436067134 1loss/SRqqt4/detection_loss‚2.6064556268128483 /loss/FRt1/assignment_loss‚4.882316883872537 /loss/FRt3/assignment_loss‚2.244113068047323 0loss/SRqqt1/detection_loss‚2.335026141256094 % _timestamp‚1.7885233289720602e+09 1loss/SRqqt1/assignment_loss‚1.588385613169521 0loss/SRqqt4/assignment_loss‚2.41392804953185 /loss/FBt3/detection_loss‚3.6469616691271463 %loss/total_loss‚53.98450593261219 epoch‚1 /loss/FRt4/assignment_loss‚3.605850795904795 1loss/FBt3/assignment_loss‚0.26107244687076064 1loss/SRqqt3/detection_loss‚1.1380589068059093 _runtime‚2567.192498003 1loss/SRqqt2/assignment_loss‚2.712483981381292 /loss/FRt1/detection_loss‚2.7475593002403484 .loss/FRt4/detection_loss‚2.118015307129598 .loss/FBt2/detection_loss‚2.449082997110155 0loss/FBt2/assignment_loss‚0.4623285850846817 trainer/global_step‚1449% ½ŒV¯Ð 1loss/SRqqt2/detection_loss‚2.7155534987864285 .loss/FRt2/assignment_loss‚5.39235200881958 0loss/FBt1/assignment_loss‚0.4361360368177636 .loss/FRt2/detection_loss‚3.470706558227539 .loss/FRt3/detection_loss‚1.656569320904581 /loss/FBt1/detection_loss‚2.3704957420175727 /loss/FBt4/detection_loss‚1.2770239258783735 _step‚118 2loss/SRqqt3/assignment_loss‚1.1197596252087718 0loss/FBt4/assignment_loss‚0.3342698436067134 1loss/SRqqt4/detection_loss‚2.6064556268128483 /loss/FRt1/assignment_loss‚4.882316883872537 /loss/FRt3/assignment_loss‚2.244113068047323 0loss/SRqqt1/detection_loss‚2.335026141256094 % _timestamp‚1.7885233289720602e+09 1loss/SRqqt1/assignment_loss‚1.588385613169521 0loss/SRqqt4/assignment_loss‚2.41392804953185 /loss/FBt3/detection_loss‚3.6469616691271463 %loss/total_loss‚53.98450593261219 epoch‚1 /loss/FRt4/assignment_loss‚3.605850795904795 1loss/FBt3/assignment_loss‚0.26107244687076064 1loss/SRqqt3/detection_loss‚1.1380589068059093 _runtime‚2567.192498003 1loss/SRqqt2/assignment_loss‚2.712483981381292 /loss/FRt1/detection_loss‚2.7475593002403484 .loss/FRt4/detection_loss‚2.118015307129598 .loss/FBt2/detection_loss‚2.449082997110155 0loss/FBt2/assignment_loss‚0.4623285850846817 trainer/global_step‚1449vÝO Y‘°‚(:† ÍæêÔÀÙßø gpu.0.gpu‚23.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚102.254' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.4508 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0’à n±‚(:ã ÍæêÔº¶ú disk.drbd1260.out‚0.359375& proc.memory.availableMB‚ 9.25390625 disk.sda3.out‚ 74.03515625# disk.nvme0c0n1.in‚ 5907.14453125% disk./.usageGB‚1176.6443099975586 disk.sda2.out‚0 disk.dm-0.in‚ 5907.14453125! disk.nvme0n1.in‚ 5907.14453125 disk.sda1.out‚0# proc.memory.rssMB‚ 2917.11328125 network.recv‚847873 disk.dm-1.in‚0 disk.dm-1.out‚ 74.03515625$ memory_percent‚99.94351863861084 proc.cpu.threads‚17 network.sent‚ 2.834105e+06 disk.sda2.in‚0 disk.dm-0.out‚49067.56640625 disk.sda.in‚0 disk.sda3.in‚0* proc.memory.percent‚17.804646492004395 cpu‚25.340119087968965) disk./.usagePercent‚78.96558691427114 disk.sda.out‚ 74.03515625% disk.nvme0c0n1.out‚49067.56640625# disk.nvme0n1.out‚49067.56640625 disk.sda1.in‚0 disk.drbd1260.in‚0=%«œ²‚(:‘ ÜæêÔëÐÐ÷ gpu.0.gpu‚22.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚102.183' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚20.436600000000002 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÚÃÉj³‚(:ß ÜæêÔÛÙû* proc.memory.percent‚17.804646492004395 disk.sda1.out‚0 disk.dm-1.out‚ 74.34765625 disk.drbd1260.out‚0.359375 disk.sda2.out‚0 disk.dm-0.out‚49364.67578125 disk.sda3.in‚0 network.sent‚ 2.93219e+06) disk./.usagePercent‚78.96558691427114% disk./.usageGB‚1176.6443099975586% disk.nvme0c0n1.out‚49364.67578125 disk.drbd1260.in‚0 disk.sda.in‚0$ memory_percent‚99.94206428527832% proc.memory.availableMB‚ 9.4921875 network.recv‚854583# disk.nvme0n1.out‚49364.67578125 disk.sda1.in‚0 disk.sda.out‚ 74.34765625 disk.sda3.out‚ 74.34765625# proc.memory.rssMB‚ 2917.11328125 cpu‚25.3399618898368 proc.cpu.threads‚17# disk.nvme0c0n1.in‚ 5907.14453125! disk.nvme0n1.in‚ 5907.14453125 disk.dm-1.in‚0 disk.sda2.in‚0 disk.dm-0.in‚ 5907.14453125^¢T_´‚(:„ ëæêÔ Žƒö gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚92.813' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.5626 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0I/éuµ‚(:ê ëæêÔÙÊø disk.drbd1260.out‚ 0.36328125 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 5907.14453125 disk.sda3.in‚0# disk.nvme0c0n1.in‚ 5907.14453125( disk.nvme0c0n1.out‚49423.06787109375 disk.sda1.in‚0 network.sent‚ 2.937203e+06% disk./.usageGB‚1176.6443099975586# disk.dm-0.out‚49423.06787109375 disk.sda.in‚0! disk.nvme0n1.in‚ 5907.14453125 disk.sda1.out‚0 disk.dm-1.in‚0$ memory_percent‚99.94068145751953 disk.drbd1260.in‚0 disk.sda.out‚ 74.76953125 disk.sda3.out‚ 74.76953125& disk.nvme0n1.out‚49423.06787109375 disk.dm-1.out‚ 74.76953125# proc.memory.availableMB‚9.71875 cpu‚25.33971297569923 network.recv‚858113) disk./.usagePercent‚78.96558691427114# proc.memory.rssMB‚ 2917.11328125* proc.memory.percent‚17.804646492004395 proc.cpu.threads‚17å½}û¶‚(:… úæêÔ­í­ö gpu.0.gpu‚5.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚116.102' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.2204 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Á€nžr·‚(:ç úæêÔÏÌÒø network.recv‚861799 disk.sda2.in‚0# disk.dm-0.out‚49716.23974609375 disk.sda.in‚0 disk.sda.out‚ 75.1796875 disk.dm-1.in‚0 disk.dm-1.out‚ 75.1796875) disk./.usagePercent‚78.96559459449716 disk.sda2.out‚0( disk.nvme0c0n1.out‚49716.23974609375 disk.drbd1260.in‚0& proc.memory.availableMB‚ 9.71484375# proc.memory.rssMB‚ 2917.11328125 proc.cpu.threads‚17% disk./.usageGB‚1176.6444244384766 disk.dm-0.in‚ 6046.5703125 disk.sda3.in‚0" disk.nvme0c0n1.in‚ 6046.5703125 disk.nvme0n1.in‚ 6046.5703125& disk.nvme0n1.out‚49716.23974609375 disk.sda1.out‚0$ memory_percent‚99.94070529937744 network.sent‚ 2.942292e+06 disk.sda3.out‚ 75.1796875 disk.sda1.in‚0 disk.drbd1260.out‚ 0.36328125* proc.memory.percent‚17.804646492004395 cpu‚25.33938668105758» Nœ¸‚(:‘ ‰çêÔÄü…÷ gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚113.113' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.622600000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þÜôüt¹‚(:é ‰çêÔ¤ù network.recv‚866475) disk./.usagePercent‚78.92344730611259$ disk./.usageGB‚1176.016399383545 disk.sda2.out‚0 disk.dm-0.in‚ 6046.5703125 disk.sda3.in‚0 disk.dm-1.out‚ 75.59765625 disk.sda2.in‚0 disk.sda.out‚ 75.59765625 disk.sda3.out‚ 75.59765625 disk.drbd1260.out‚ 0.3671875$ memory_percent‚99.94075298309326# proc.memory.rssMB‚ 2917.11328125* proc.memory.percent‚17.804646492004395 cpu‚25.337947919254674" disk.nvme0c0n1.in‚ 6046.5703125 disk.sda.in‚0 disk.nvme0n1.in‚ 6046.5703125& disk.nvme0n1.out‚50188.68603515625 disk.sda1.out‚0 disk.dm-1.in‚0 disk.drbd1260.in‚0 proc.cpu.threads‚17# disk.dm-0.out‚50188.68603515625( disk.nvme0c0n1.out‚50188.68603515625 disk.sda1.in‚0& proc.memory.availableMB‚ 9.70703125 network.sent‚ 2.948303e+06¡%‹‡2º- _wandbruntime‚2641  _runtime‚2641TÈ…Æ»À & lr-AdamW/pg1‚0.0009172715550246138 & lr-AdamW/pg2‚0.0009172715550246138 trainer/global_step‚1499 % _timestamp‚1.7885234028216274e+09 _runtime‚ 2641.04204814 _step‚119›½(Ê¼Ä & lr-AdamW/pg2‚0.0009172715550246138 trainer/global_step‚1499 % _timestamp‚1.7885234028216274e+09 _runtime‚ 2641.04204814 _step‚119 & lr-AdamW/pg1‚0.0009172715550246138wZäá2½- _wandbruntime‚2641  _runtime‚2641—p…´¾ÂO SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚60840944774144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3028974603776 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚60886465853765 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚69083579 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚67485132077056 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16947686 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚672 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 [IDeviceStatsMonitor.on’+T„"_train_batch_start/active_bytes.small_pool.allocated‚ 2975502675456 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚28394057 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28394055 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚60840926771200 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2975269007872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚40715866 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27729288 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚40689524 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚60886219833329 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚40715866 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚69109923 trainer/global_step‚1499 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚63816447449600 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚64455845009920 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚64455868950016 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288523264 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚57919400044016 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10780932 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚57919383004656 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚60840926771200 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2975502675456 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28394057 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3029263127040 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚2641.043493645 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2967065809749 % _timestamp‚1.7885234028237705e+09 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚63816447449600 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚63816195779072 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2975269007872 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚40689524 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312463360 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚67484819613696 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚63816195779072 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28394055 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0$ÝþÌ'¿ÆO H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10780932 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚57919383004656 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚60840926771200 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2975502675456 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28394057 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3029263127040 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚2641.043493645 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2967065809749 % _timestamp‚1.7885234028237705e+09 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚63816447449600 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚63816195779072 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2975269007872 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚40689524 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312463360 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚67484819613696 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚63816195779072 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28394055 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚60840944774144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3028974603776 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚60886465853765 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚69083579 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚67485132077056 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16947686 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚672 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2975502675456 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚28394057 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28394055 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚60840926771200 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2975269007872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚40715866 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27729288 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚40689524 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚60886219833329 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚40715866 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚69109923 trainer/global_step‚1499 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚63816447449600 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚64455845009920 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚64455868950016 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288523264 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚57919400044016 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 IDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27746273 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚57956913362080 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10787787 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 trainer/global_step‚1499 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚63857697525760 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚67528808720896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚63857697525760 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚40742584 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2977466714624 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2977466714624 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚69154916 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚69154916 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27746954 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚60925954700383 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 _runtime‚2642.092281996 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28412330 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚64497552920064 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28412332 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚121 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚40742584 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚67528494173696 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚57956930401440 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2969024298943 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚64497528979968 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚60880212808192 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3030965193728 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290607104 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28412332 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16958488 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚60880230811136 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚69128546 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314547200 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360¢Þ½ùÂÐM I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚63857697525760 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚67528808720896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚63857697525760 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚40742584 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2977466714624 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2977466714624 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚69154916 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚69154916 OBDeviceStatsMonitor.on_train_batch_end/inactive_spli Ö»Ýt.all.allocated‚27746954 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚60925954700383 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 _runtime‚2642.092281996 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28412330 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚64497552920064 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28412332 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚121 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚40742584 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚67528494173696 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚57956930401440 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2969024298943 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚64497528979968 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚60880212808192 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3030965193728 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290607104 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28412332 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16958488 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚60880230811136 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚69128546 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314547200 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27746273 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚57956913362080 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10787787 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 trainer/global_step‚1499y-bIâ2Ã- _wandbruntime‚2642  _runtime‚2642œ›^YÄÓ 1loss/SRqqt1/detection_loss‚2.3106631789888654 1loss/SRqqt3/detection_loss‚1.0725538418960892 1loss/SRqqt4/detection_loss‚3.2962460198572705 epoch‚1 .loss/FBt3/detection_loss‚4.329351842403412 _step‚122 1loss/FBt4/assignment_loss‚0.27667935511496466 .loss/FBt2/detection_loss‚2.198528003124964 &loss/total_loss‚54.812582967935164 $ _timestamp‚1.788523403875519e+09 1loss/SRqqt2/assignment_loss‚2.690945542520947 0loss/FBt1/assignment_loss‚0.3039122035342563 /loss/FRt2/detection_loss‚3.2187213039398195 1loss/SRqqt2/detection_loss‚2.8679004543357425 trainer/global_step‚1499 /loss/FRt1/assignment_loss‚4.448634839827014 /loss/FRt1/detection_loss‚2.8243158881208905 /loss/FRt2/assignment_loss‚5.664450225830078 .loss/FRt3/assignment_loss‚2.22179514169693 /loss/FRt4/assignment_loss‚3.286404313994389 2loss/SRqqt1/assignment_loss‚1.6386364017214095 2loss/SRqqt4/assignment_loss‚2.8003875953810557 0loss/FBt2/assignment_loss‚0.3815476477944425 1loss/FBt3/assignment_loss‚0.08236226439473171 /loss/FBt4/detection_loss‚0.8921141205282765 2loss/SRqqt3/assignment_loss‚1.4883750343163098 /loss/FRt3/detection_loss‚1.8240909222258281 /loss/FRt4/detection_loss‚1.9958219995685653 /loss/FBt1/detection_loss‚2.6981448268189148 _runtime‚2642.0957025547îo]Å× 1loss/FBt4/assignment_loss‚0.27667935511496466 .loss/FBt2/detection_loss‚2.198528003124964 &loss/total_loss‚54.812582967935164 $ _timestamp‚1.788523403875519e+09 1loss/SRqqt2/assignment_loss‚2.690945542520947 0loss/FBt1/assignment_loss‚0.3039122035342563 /loss/FRt2/detection_loss‚3.2187213039398195 1loss/SRqqt2/detection_loss‚2.8679004543357425 trainer/global_step‚1499 /loss/FRt1/assignment_loss‚4.448634839827014 /loss/FRt1/detection_loss‚2.8243158881208905 /loss/FRt2/assignment_loss‚5.664450225830078 .loss/FRt3/assignment_loss‚2.22179514169693 /loss/FRt4/assignment_loss‚3.286404313994389 2loss/SRqqt1/assignment_loss‚1.6386364017214095 2loss/SRqqt4/assignment_loss‚2.8003875953810557 0loss/FBt2/assignment_loss‚0.3815476477944425 1loss/FBt3/assignment_loss‚0.08236226439473171 /loss/FBt4/detection_loss‚0.8921141205282765 2loss/SRqqt3/assignment_loss‚1.4883750343163098 /loss/FRt3/detection_loss‚1.8240909222258281 /loss/FRt4/detection_loss‚1.9958219995685653 /loss/FBt1/detection_loss‚2.6981448268189148 _runtime‚2642.095702554 1loss/SRqqt1/detection_loss‚2.3106631789888654 1loss/SRqqt3/detection_loss‚1.0725538418960892 1loss/SRqqt4/detection_loss‚3.2962460198572705 epoch‚1 .loss/FBt3/detection_loss‚4.329351842403412 _step‚122z¹˜6‘Æ‚(:† ˜çêÔÒÏ ÷ gpu.0.gpu‚11.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚103.482' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.6964 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ïò¸ÈsÇ‚(:è ˜çêÔ¦ÄÀ÷ disk.sda.out‚ 75.98046875 disk.dm-1.out‚ 75.98046875 disk.drbd1260.out‚ 0.3671875& proc.memory.availableMB‚ 9.65234375# proc.memory.rssMB‚ 2917.11328125* proc.memory.percent‚17.804646492004395 network.recv‚871326% disk./.usageGB‚1175.9929313659668 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 6046.5703125& disk.nvme0n1.out‚50471.91259765625 disk.dm-1.in‚0" memory_percent‚99.941086769104 proc.cpu.threads‚17 disk.dm-0.in‚ 6046.5703125 disk.sda.in‚0( disk.nvme0c0n1.out‚50471.91259765625 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0# disk.dm-0.out‚50471.91259765625 disk.sda3.in‚0 disk.sda3.out‚ 75.98046875 disk.nvme0n1.in‚ 6046.5703125 disk.drbd1260.in‚0 cpu‚25.336754795442797 network.sent‚ 2.954529e+06) disk./.usagePercent‚78.92187234776195+ 1àÈ‚(:… §çêÔ°’Ï÷ gpu.0.gpu‚25.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚93.913' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.7826 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0VnòºsÉ‚(:è §çêÔŒ”÷ú" disk.nvme0c0n1.in‚ 6046.5703125& disk.nvme0n1.out‚50812.96142578125 disk.sda2.in‚0# disk.dm-0.out‚50812.96142578125 disk.sda.out‚ 76.3671875 disk.sda3.in‚0$ memory_percent‚99.93956089019775& proc.memory.availableMB‚ 9.90234375 disk.dm-1.in‚0 disk.dm-1.out‚ 76.3671875 disk.sda3.out‚ 76.3671875* proc.memory.percent‚17.804646492004395 cpu‚25.335567262986952 network.recv‚876606% disk./.usageGB‚1175.9929542541504 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda.in‚0# proc.memory.rssMB‚ 2917.11328125 proc.cpu.threads‚17 network.sent‚ 3.061967e+06) disk./.usagePercent‚78.92187388380715( disk.nvme0c0n1.out‚50812.96142578125 disk.nvme0n1.in‚ 6046.5703125 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.37109375 disk.dm-0.in‚ 6046.5703125½x£‘Ê‚(:† ¶çêÔÈÂÍö gpu.0.gpu‚25.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚104.952' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.9904 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0X …yË‚(:î ¶çêÔ ¯³÷( disk.nvme0c0n1.out‚50976.05908203125 disk.sda1.in‚0 disk.sda1.out‚0 cpu‚25.334617750006537 proc.cpu.threads‚17 network.sent‚ 3.066138e+06 disk.sda.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 76.85546875 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.37109375 disk.dm-0.in‚ 6218.50390625) disk./.usagePercent‚78.92344269797698 disk.sda.out‚ 76.85546875# disk.nvme0c0n1.in‚ 6218.50390625& disk.nvme0n1.out‚50976.05908203125 disk.dm-1.out‚ 76.85546875 disk.sda2.out‚0# disk.dm-0.out‚50976.05908203125$ memory_percent‚99.93960857391357 network.recv‚879413! disk.nvme0n1.in‚ 6218.50390625 disk.sda2.in‚0& proc.memory.availableMB‚ 9.89453125# proc.memory.rssMB‚ 2917.11328125* proc.memory.percent‚17.804646492004395% disk./.usageGB‚1176.0163307189941˜‰Ì‚(:„ ÅçêÔ­™¤÷ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚29.0 gpu.0.powerWatts‚92.922' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.5844 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Aõ•mÍ‚(:â ÅçêÔÑüÈû) disk./.usagePercent‚78.92344320999204 disk.sda1.in‚0 disk.dm-1.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 6218.50390625 disk.sda.out‚ 77.33203125% proc.memory.availableMB‚ 9.8828125# proc.memory.rssMB‚ 2917.11328125 network.sent‚ 3.07213e+06% disk./.usageGB‚1176.0163383483887# disk.nvme0c0n1.in‚ 6218.50390625" disk.dm-0.out‚51124.9111328125 disk.sda3.in‚0# memory_percent‚99.9396800994873* proc.memory.percent‚17.804646492004395' disk.nvme0c0n1.out‚51124.9111328125% disk.nvme0n1.out‚51124.9111328125 disk.drbd1260.out‚0.375 cpu‚25.33453133811371 proc.cpu.threads‚17 network.recv‚883859! disk.nvme0n1.in‚ 6218.50390625 disk.sda1.out‚0 disk.dm-1.out‚ 77.33203125 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.sda.in‚0 disk.sda3.out‚ 77.332031258ë»z2Î- _wandbruntime‚2712  _runtime‚27123«ëÆÏÀ _runtime‚ 2712.00514919 _step‚123 & lr-AdamW/pg1‚0.0009166691766655371 & lr-AdamW/pg2‚0.0009166691766655371 trainer/global_step‚1549 % _timestamp‚1.7885234737852802e+09Û:}4ÊÐÄ _runtime‚ 2712.00514919 _step‚123 & lr-AdamW/pg1‚0.0009166691766655371 & lr-AdamW/pg2‚0.0009166691766655371 trainer/global_step‚1549 % _timestamp‚1.7885234737852802e+09{ ›‰Y2Ñ- _wandbruntime‚2712  _runtime‚2712߬ïÂÇ'ÒÁO WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚71361625 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚676 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3128784711680 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3065003850257 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚29307807 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17487786 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292717568 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚65878713335296 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚62805246623744 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28616899 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚62805246623744 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚2712.006674105 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3073484714496 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚42053818 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚674 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚65878713335296 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚62805228620800 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3073718382080 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚65878965005824 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚42027476 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚28617575 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11129789 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚42027476 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚62860921765473 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 trainer/global_step‚1549 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11129115A‰‰”õÓÅO SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11129115 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚71361625 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚676 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3128784711680 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3065003850257 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚29307807 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17487786 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292717568 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚65878713335296 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚62805246623744 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28616899 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚62805246623744 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚2712.006674105 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3073484714496 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚42053818 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚674 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚65878713335296 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚62805228620800 ^LDeŸJÈþÖviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3073718382080 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚65878965005824 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚42027476 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚28617575 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11129789 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚42027476 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚62860921765473 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 trainer/global_step‚1549|<"†2Ô- _wandbruntime‚2712  _runtime‚2712|eáÒ&ÕÌM OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚29326080 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚71406618 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚678 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3075448740352 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚29326082 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚42054168 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚62844514657792 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11136132 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚65920215081984 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚69712820550144 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚66581727482368 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3066962339451 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28635396 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚71380248 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 288509952 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚62844532660736 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚62844532660736 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚59833431233280 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3075682421248 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚676 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 % _timestamp‚1.7885234747126567e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17498588 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 312450048 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚29326082 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚65920215081984 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚62900410612091 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚65919963398144 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3075682421248 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 _runtime‚2712.933530216 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚42080536 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚71406618 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3075448740352 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚66581751422464 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3131069127680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3130780617728 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 trainer/global_step‚1549 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚42080536 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28634718 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11136808 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚69712508100096 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17498586 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚59833448272640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 _step‚125 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚71380248 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚42054168 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0sÆ„íÖ&ÖÐM S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚65920215081984 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚62900410612091 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚65919963398144 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3075682421248 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 _runtime‚2712.933530216 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚42080536 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚71406618 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3075448740352 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚66581751422464 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3131069127680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3130780617728 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 trainer/global_step‚1549 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚42080536 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28634718 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11136808 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚69712508100096 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17498586 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚59833448272640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 _step‚125 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚71380248 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚42054168 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚29326080 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚71406618 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚678 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3075448740352 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚29326082 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚42054168 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚62844514657792 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11136132 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚65920215081984 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚69712820550144 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚66581727482368 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3066962339451 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28635396 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚71380248 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 288509952 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚62844532660736 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚62844532660736 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚59833431233280 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3075682421248 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚676 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 % _timestamp‚1.7885234747126567e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17498588 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 312450048 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚29326082 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582}Hu»Ì2×- _wandbruntime‚2712  _runtime‚2712• lsTØÎ /loss/FRt2/assignment_loss‚5.183171045212519 /loss/FRt3/assignment_loss‚2.254332238435745 %loss/total_loss‚52.59738203927573 2loss/SRqqt1/assignment_loss‚1.9273093238070205 1loss/FBt1/assignment_loss‚0.26675812512951325 0loss/FBt3/assignment_loss‚0.3442927300644938 /loss/FRt3/detection_loss‚1.7787717139348387 1loss/SRqqt1/detection_loss‚2.2112667326991624 /loss/FRt4/assignment_loss‚3.480434167385101 1loss/SRqqt2/assignment_loss‚1.951479048891501 1loss/SRqqt3/detection_loss‚1.1393446446142412 /loss/FBt2/detection_loss‚2.5319249944554434 trainer/global_step‚1549 2loss/SRqqt3/assignment_loss‚1.4769564088433982 1loss/SRqqt4/assignment_loss‚2.168497156013142 -loss/FRt4/detection_loss‚2.06384250536561 1loss/SRqqt2/detection_loss‚3.0913667570460928 .loss/FBt3/detection_loss‚3.010518617100186 _runtime‚ 2712.93477858 0loss/FRt1/assignment_loss‚3.9966314826692853 .loss/FRt1/detection_loss‚2.611127824442727 .loss/FRt2/detection_loss‚3.402775775818598 .loss/FBt1/detection_loss‚2.462761238217354 _step‚126 0loss/FBt2/assignment_loss‚0.4530120913049258 0loss/SRqqt4/detection_loss‚3.599939926104112 /loss/FBt4/detection_loss‚0.8463131672358268 % _timestamp‚1.7885234747152772e+09 1loss/FBt4/assignment_loss‚0.34455432448489215 epoch‚1Q@XÙÒ 2loss/SRqqt1/assignment_loss‚1.9273093238070205 1loss/FBt1/assignment_loss‚0.26675812512951325 0loss/FBt3/assignment_loss‚0.3442927300644938 /loss/FRt3/detection_loss‚1.7787717139348387 1loss/SRqqt1/detection_loss‚2.2112667326991624 /loss/FRt4/assignment_loss‚3.480434167385101 1loss/SRqqt2/assignment_loss‚1.951479048891501 1loss/SRqqt3/detection_loss‚1.1393446446142412 /loss/FBt2/detection_loss‚2.5319249944554434 trainer/global_step‚1549 2loss/SRqqt3/assignment_loss‚1.4769564088433982 1loss/SRqqt4/assignment_loss‚2.168497156013142 -loss/FRt4/detection_loss‚2.06384250536561 1loss/SRqqt2/detection_loss‚3.0913667570460928 .loss/FBt3/detection_loss‚3.010518617100186 _runtime‚ 2712.93477858 0loss/FRt1/assignment_loss‚3.9966314826692853 .loss/FRt1/detection_loss‚2.611127824442727 .loss/FRt2/detection_loss‚3.402775775818598 .loss/FBt1/detection_loss‚2.462761238217354 _step‚126 0loss/FBt2/assignment_loss‚0.4530120913049258 0loss/SRqqt4/detection_loss‚3.599939926104112 /loss/FBt4/detection_loss‚0.8463131672358268 % _timestamp‚1.7885234747152772e+09 1loss/FBt4/assignment_loss‚0.34455432448489215 epoch‚1 /loss/FRt2/assignment_loss‚5.183171045212519 /loss/FRt3/assignment_loss‚2.254332238435745 %loss/total_loss‚52.59738203927573~Xö㨑Ú‚(:† ÔçêÔØ´²ö gpu.0.gpu‚19.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚143.982' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.7964 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0|ìònÛ‚(:ã ÔçêÔ§ûß÷) disk./.usagePercent‚78.91515240599567 disk.drbd1260.in‚0 disk.sda2.in‚0$ memory_percent‚99.94142055511475 cpu‚25.333422348401534 proc.cpu.threads‚17 disk.drbd1260.out‚0.375 network.recv‚888487% disk./.usageGB‚1175.8927993774414' disk.nvme0c0n1.out‚51265.0322265625 disk.sda1.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 77.8515625 disk.sda.in‚0 network.sent‚ 3.078211e+06! disk.nvme0n1.in‚ 6218.50390625% disk.nvme0n1.out‚51265.0322265625 disk.dm-0.in‚ 6218.50390625 disk.sda.out‚ 77.8515625& proc.memory.availableMB‚ 9.59765625# proc.memory.rssMB‚ 2917.11328125* proc.memory.percent‚17.804646492004395# disk.nvme0c0n1.in‚ 6218.50390625 disk.sda1.in‚0 disk.sda2.out‚0" disk.dm-0.out‚51265.0322265625 disk.sda3.in‚0 disk.sda3.out‚ 77.8515625¾`Mg‘Ü‚(:† ãçêÔï¤Í÷ gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚110.993' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.1986 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ûÝe(kÝ‚(:à ãçêÔ“ã†û disk.sda1.in‚0 disk.sda2.in‚0" disk.dm-0.out‚51610.6513671875$ memory_percent‚99.94077682495117$ proc.memory.availableMB‚9.703125 network.recv‚893748 disk.dm-1.out‚ 78.234375 disk.drbd1260.in‚0 disk.sda3.out‚ 78.234375* proc.memory.percent‚17.804670333862305 proc.cpu.threads‚17 network.sent‚ 3.185563e+06# disk.nvme0c0n1.in‚ 6218.50390625' disk.nvme0c0n1.out‚51610.6513671875% disk.nvme0n1.out‚51610.6513671875 disk.sda1.out‚0 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.37890625 disk.dm-0.in‚ 6218.50390625) disk./.usagePercent‚78.92043640150239$ disk./.usageGB‚1175.971534729004 disk.sda2.out‚0 disk.sda.in‚0 disk.sda.out‚ 78.234375 disk.sda3.in‚0" proc.memory.rssMB‚ 2917.1171875 cpu‚25.33219075440792! disk.nvme0n1.in‚ 6218.50390625Ö ÚXÞ‚(:‚ òçêÔäúÑö gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.14' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.628 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0$Öéoß‚(:ä òçêÔ®®šù disk.dm-0.in‚ 6344.27734375 network.sent‚ 3.190606e+06 network.recv‚897238 disk.dm-1.out‚ 78.703125 disk.drbd1260.in‚0 disk.sda2.out‚0 disk.sda3.in‚0' disk.nvme0c0n1.out‚51930.4677734375" disk.dm-0.out‚51930.4677734375& proc.memory.availableMB‚ 9.69921875* proc.memory.percent‚17.804670333862305 cpu‚25.331069504431348 proc.cpu.threads‚17 disk.sda.in‚0 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.37890625 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 6344.27734375! disk.nvme0n1.in‚ 6344.27734375$ memory_percent‚99.94080066680908" proc.memory.rssMB‚ 2917.1171875) disk./.usagePercent‚78.92730405961592% disk./.usageGB‚1176.0738677978516 disk.sda.out‚ 78.703125 disk.sda3.out‚ 78.703125% disk.nvme0n1.out‚51930.4677734375 disk.sda1.in‚0 disk.sda1.out‚0O7Õ‘à‚(:† èêÔøî…÷ gpu.0.gpu‚21.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚104.547' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.9094 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Y«Åuá‚(:ê èêÔë„îø network.sent‚ 3.196586e+06) disk./.usagePercent‚78.92572910126528! disk.nvme0n1.in‚ 6344.27734375& disk.nvme0n1.out‚52099.23291015625 disk.sda1.out‚0$ memory_percent‚99.94091987609863" proc.memory.rssMB‚ 2917.1171875 network.recv‚901725( disk.nvme0c0n1.out‚52099.23291015625 disk.dm-1.in‚0 disk.dm-1.out‚ 79.48046875 disk.drbd1260.out‚ 0.3828125 disk.sda2.in‚0 disk.sda3.in‚0* proc.memory.percent‚17.804670333862305 disk.sda1.in‚0# disk.dm-0.out‚52099.23291015625 disk.sda.in‚0 disk.sda.out‚ 79.48046875 disk.sda3.out‚ 79.48046875% proc.memory.availableMB‚ 9.6796875 proc.cpu.threads‚17% disk./.usageGB‚1176.0503997802734# disk.nvme0c0n1.in‚ 6344.27734375 disk.drbd1260.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 6344.27734375 cpu‚25.33091025334854ˆâ‚(:„ èêԤ˗÷ gpu.0.gpu‚19.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚101.72' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.344 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0.ãUÿtã‚(:é èêÔøŒ°ù disk.sda1.in‚0 disk.drbd1260.in‚0 cpu‚25.330117257339243 proc.cpu.threads‚17% disk./.usageGB‚1176.0785942077637! disk.nvme0n1.in‚ 6344.27734375& disk.nvme0n1.out‚52655.79541015625 disk.sda2.in‚0 disk.sda3.out‚ 79.8203125# disk.nvme0c0n1.in‚ 6344.27734375$ memory_percent‚99.94251728057861# proc.memory.rssMB‚ 2917.12109375 network.sent‚ 3.20276e+06 network.recv‚906233 disk.dm-1.in‚0 disk.dm-0.in‚ 6344.27734375# disk.dm-0.out‚52655.79541015625 disk.sda.in‚0 disk.sda.out‚ 79.8203125* proc.memory.percent‚17.804694175720215) disk./.usagePercent‚78.92762125295084 disk.sda1.out‚0 disk.dm-1.out‚ 79.8203125 disk.drbd1260.out‚ 0.3828125 disk.sda2.out‚0 disk.sda3.in‚0( disk.nvme0c0n1.out‚52655.79541015625& proc.memory.availableMB‚ 9.41796875{¹(2ä- _wandbruntime‚2788  _runtime‚2788¢ÈfšÆåÀ _step‚127 & lr-AdamW/pg1‚0.0009159715125462031 & lr-AdamW/pg2‚0.0009159715125462031 trainer/global_step‚1599 $ _timestamp‚1.788523550054306e+09 _runtime‚2788.274963038}„ÆäÊæÄ _step‚127 & lr-AdamW/pg1‚0.0009159715125462031 & lr-AdamW/pg2‚0.0009159715125462031 trainer/global_step‚1599 $ _timestamp‚1.788523550054306e+09 _runtime‚2788.274963038o b2ç- _wandbruntime‚2788  _runtime‚2788|y"äèÂO L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚30221555 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3171934088704 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11479009 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 trainer/global_step‚1599 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 D>DeviceStatsMonitorþ‹ä$.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚64769548473344 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 _runtime‚2788.278663795 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 314560512 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3171934088704 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚43391770 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚29506893 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚71853075255296 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚43391770 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11479716 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 % _timestamp‚1.7885235500574589e+09 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚73586983 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 _step‚128 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚709 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 290620416 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚61672418747056 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3171700421120 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3228809300480 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚67941230891520 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3171700421120 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚29507602 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚30221557 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18027884 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚68624265954816 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚67941482562048 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18027886 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚64835131656745 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚61672435786416 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3162712909689 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚64769530470400 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30221555 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚43365428 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚67941482562048 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚43391770 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚29506893 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚71853075255296 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚43391770 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11479716 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 % _timestamp‚1.7885235500574589e+09 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚73586983 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 _step‚128 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚709 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 290620416 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚61672418747056 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3171700421120 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3228809300480 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚67941230891520 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3171700421120 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚29507602 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚30221557 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18027884 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚68624265954816 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚67941482562048 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18027886 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚64835131656745 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚61672435786416 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3162712909689 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚64769530470400 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30221555 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚43365428 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚67941482562048 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚64769548473344 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 _runtime‚2788.278663795€š~ˆ`›ê‚(: ŸèêÔÓ§‹÷ gpu.0.gpu‚25.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚97.146' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚19.429199999999998 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÄSçrë‚(:ç ŸèêÔÇ£×ù disk.sda2.in‚0 disk.sda3.out‚ 80.2109375 disk.dm-1.in‚0 disk.drbd1260.in‚0# proc.memory.rssMB‚ 2917.12890625 proc.cpu.threads‚17" disk.dm-0.out‚52867.0302734375! disk.nvme0n1.in‚ 6344.27734375% disk.nvme0n1.out‚52867.0302734375 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚99.94425773620605% disk./.usageGB‚1176.0785942077637 disk.sda.out‚ 80.2109375' disk.nvme0c0n1.out‚52867.0302734375 disk.drbd1260.out‚ 0.38671875% proc.memory.availableMB‚ 9.1328125* proc.memory.percent‚17.804741859436035 cpu‚25.330040476631147 network.sent‚ 3.207715e+06) disk./.usagePercent‚78.92762125295084 disk.sda2.out‚0 disk.dm-0.in‚ 6344.27734375 disk.sda.in‚0 disk.sda3.in‚0# disk.nvme0c0n1.in‚ 6344.27734375 disk.dm-1.out‚ 80.2109375 network.recv‚909465:™Ÿœ2ì- _wandbruntime‚2790  _runtime‚2790[Oå£Ñ&íËM C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3230804247040 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚30239832 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚43418488 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7885235522672958e+09 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚61709949104480 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚73631950 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚73658320 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 trainer/global_step‚1599 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚30239830 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3173898127872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3173898127872 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18038688 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚67982480954368 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚43392120 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3230507348480 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29524403 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚68665925984768 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚64874620503155 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚73631950 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚30239830 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚71896754171904 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚68665949924864 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 _runtime‚ 2790.48957695 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚67982732638208 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚64874866523799 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚677 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 320838656 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚64808816507392 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2òß4ÐîÐM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29524403 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚68665925984768 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚64874620503155 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚73631950 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚30239830 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚71896754171904 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚68665949924864 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 _runtime‚ 2790.48957695 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚67982732638208 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚64874866523799 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 9b°È565024768 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚677 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 320838656 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚64808816507392 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3230804247040 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚30239832 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚43418488 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7885235522672958e+09 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚61709949104480 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚73631950 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚73658320 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 trainer/global_step‚1599 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚30239830 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3173898127872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3173898127872 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18038688 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚67982480954368 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚43392120 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3230507348480NîÖ2ï- _wandbruntime‚2790  _runtime‚2790\â\ö[ðÕ /loss/FRt3/detection_loss‚1.8266098256553374 1loss/SRqqt2/detection_loss‚2.6330181317670003 _runtime‚2790.492700819 1loss/SRqqt3/assignment_loss‚1.295418012758781 0loss/FBt4/assignment_loss‚0.2154861096867276 /loss/FRt1/detection_loss‚2.5850298936302596 /loss/FRt4/detection_loss‚2.0886737688754997 /loss/FRt2/assignment_loss‚4.712184559215199 1loss/SRqqt3/detection_loss‚1.1447000902081872 /loss/FBt1/detection_loss‚2.4063262436538935 $ _timestamp‚1.788523552272691e+09 2loss/SRqqt2/assignment_loss‚1.9930896333285741 2loss/SRqqt4/assignment_loss‚2.3908408737700917 1loss/FBt1/assignment_loss‚0.49096492514891565 1loss/SRqqt1/detection_loss‚2.2977482395512716 1loss/SRqqt4/detection_loss‚2.3985993369765906 /loss/FRt1/assignment_loss‚5.297719878119391 0loss/FRt4/assignment_loss‚3.0662958845496178 1loss/SRqqt1/assignment_loss‚2.024348743898528 epoch‚1 0loss/FRt3/assignment_loss‚1.7115996758784018 1loss/FBt2/assignment_loss‚0.24040421071592444 /loss/FBt4/detection_loss‚1.1365025619181386 trainer/global_step‚1599 /loss/FRt2/detection_loss‚3.1127142418514597 .loss/FBt2/detection_loss‚2.513135089593775 _step‚130 0loss/FBt3/assignment_loss‚0.3306334889493314 /loss/FBt3/detection_loss‚3.1511237025260925 %loss/total_loss‚51.06316712222699ˆût%`ñÚ 0loss/FRt4/assignment_loss‚3.0662958845496178 1loss/SRqqt1/assignment_loss‚2.024348743898528 epoch‚1 0loss/FRt3/assignment_loss‚1.7115996758784018 1loss/FBt2/assignment_loss‚0.24040421071592444 /loss/FBt4/detection_loss‚1.1365025619181386 trainer/global_step‚1599 /loss/FRt2/detection_loss‚3.1127142418514597 .loss/FBt2/detection_loss‚2.513135089593775 _step‚130 0loss/FBt3/assignment_loss‚0.3306334889493314 /loss/FBt3/detection_loss‚3.1511237025260925 %loss/total_loss‚51.06316712222699 /loss/FRt3/detection_loss‚1.8266098256553374 1loss/SRqqt2/detection_loss‚2.6330181317670003 _runtime‚2790.492700819 1loss/SRqqt3/assignment_loss‚1.295418012758781 0loss/FBt4/assignment_loss‚0.2154861096867276 /loss/FRt1/detection_loss‚2.5850298936302596 /loss/FRt4/detection_loss‚2.0886737688754997 /loss/FRt2/assignment_loss‚4.712184559215199 1loss/SRqqt3/detection_loss‚1.1447000902081872 /loss/FBt1/detection_loss‚2.4063262436538935 $ _timestamp‚1.788523552272691e+09 2loss/SRqqt2/assignment_loss‚1.9930896333285741 2loss/SRqqt4/assignment_loss‚2.3908408737700917 1loss/FBt1/assignment_loss‚0.49096492514891565 1loss/SRqqt1/detection_loss‚2.2977482395512716 1loss/SRqqt4/detection_loss‚2.3985993369765906 /loss/FRt1/assignment_loss‚5.297719878119391‚Þe&ò‚(:‚ ®èêÔðÁ¹÷ gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.14' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.628 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0”ë@ntó‚(:é ®èêÔ£”†ú$ memory_percent‚99.94304180145264* proc.memory.percent‚17.804741859436035 network.recv‚914703$ disk./.usageGB‚1176.141300201416 disk.sda2.out‚0 disk.dm-0.in‚ 6532.01171875 disk.sda.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 80.48046875% disk.nvme0n1.out‚52917.6630859375 network.sent‚ 3.30388e+06 disk.drbd1260.out‚ 0.38671875# disk.nvme0c0n1.in‚ 6532.01171875 disk.sda1.out‚0 disk.dm-1.in‚0& proc.memory.availableMB‚ 9.33203125# proc.memory.rssMB‚ 2917.12890625 cpu‚25.329875909995675 disk.drbd1260.in‚0 disk.sda2.in‚0" disk.dm-0.out‚52886.9130859375! disk.nvme0n1.in‚ 6532.01171875 proc.cpu.threads‚17) disk./.usagePercent‚78.93182950479869 disk.sda.out‚ 80.48046875' disk.nvme0c0n1.out‚52917.4130859375 disk.sda1.in‚0 disk.dm-1.out‚ 80.480468754©Í ‘ô‚(:† ½èêÔªÍçö gpu.0.gpu‚15.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚103.316' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.6632 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ä.ô™võ‚(:ë ½èêÔ™öø! disk.nvme0n1.in‚ 6532.01171875 disk.sda1.in‚0 disk.dm-1.out‚ 81.05078125 network.sent‚ 3.401819e+06 disk.drbd1260.out‚0.390625 disk.sda.out‚ 81.05078125$ memory_percent‚99.94299411773682# proc.memory.rssMB‚ 2917.12890625 cpu‚25.329757195179194 network.recv‚921073 disk.drbd1260.in‚0 disk.sda.in‚0# disk.nvme0c0n1.in‚ 6532.01171875& disk.nvme0n1.out‚53164.86083984375& proc.memory.availableMB‚ 9.33984375 disk.sda2.in‚0 disk.dm-0.in‚ 6532.01171875# disk.dm-0.out‚53164.86083984375 disk.sda3.out‚ 81.05078125 disk.sda1.out‚0 disk.dm-1.in‚0* proc.memory.percent‚17.804741859436035 proc.cpu.threads‚17) disk./.usagePercent‚78.93444897388903$ disk./.usageGB‚1176.180332183838 disk.sda2.out‚0 disk.sda3.in‚0( disk.nvme0c0n1.out‚53164.86083984375âŒÞvö‚(:ë ÌèêÔÒèïù disk.sda.in‚0 disk.sda2.out‚0 disk.sda.out‚ 81.37890625# disk.nvme0c0n1.in‚ 6532.01171875( disk.nvme0c0n1.out‚53415.47412109375# memory_percent‚99.9427080154419& proc.memory.availableMB‚ 9.38671875# proc.memory.rssMB‚ 2917.12890625 proc.cpu.threads‚17 disk.sda2.in‚0 network.sent‚ 3.407891e+06) disk./.usagePercent‚78.93437985185479 disk.sda1.in‚0 disk.sda3.out‚ 81.37890625! disk.nvme0n1.in‚ 6532.01171875& disk.nvme0n1.out‚53415.47412109375* proc.memory.percent‚17.804741859436035 disk.dm-1.in‚0 disk.dm-1.out‚ 81.37890625 disk.sda3.in‚0 cpu‚25.329568781994272 disk.dm-0.in‚ 6532.01171875# disk.dm-0.out‚53415.47412109375 network.recv‚925673% disk./.usageGB‚1176.1793022155762 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚0.390625—¥I—‘÷‚(:† ÌèêÔ„î¦ú gpu.0.gpu‚26.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚112.874' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.5748 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0H;+ø‚(:„ ÛèêÔ•–¡ø gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚134.46' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.892 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0aFŠ}uù‚(:ê ÛèêԺ͚ù network.recv‚929097 disk.sda1.in‚0 disk.drbd1260.in‚0 disk.sda2.out‚0 cpu‚25.32951410722664 disk.drbd1260.out‚ 0.39453125) disk./.usagePercent‚78.94276870673677% disk./.usageGB‚1176.3043022155762' disk.nvme0c0n1.out‚53845.9326171875! disk.nvme0n1.in‚ 6532.01171875 disk.dm-1.in‚0 disk.sda2.in‚0 disk.dm-0.in‚ 6532.01171875 network.sent‚ 3.412892e+06 disk.sda1.out‚0 disk.sda.out‚ 81.66015625 disk.sda3.in‚0 disk.sda3.out‚ 81.66015625# proc.memory.rssMB‚ 2917.12890625* proc.memory.percent‚17.804741859436035 proc.cpu.threads‚17# disk.nvme0c0n1.in‚ 6532.01171875% disk.nvme0n1.out‚53845.9326171875 disk.dm-1.out‚ 81.66015625" disk.dm-0.out‚53845.9326171875 disk.sda.in‚0$ memory_percent‚99.94418621063232& proc.memory.availableMB‚ 9.14453125ò²ú‚(:… êèêÔã†áö gpu.0.gpu‚4.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚103.722' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.7444 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0D¨q€wû‚(:ì êèêÔž½¿÷# proc.memory.rssMB‚ 2917.12890625% disk./.usageGB‚1176.3277702331543 disk.sda2.out‚0 disk.sda.out‚ 82.03515625( disk.nvme0c0n1.out‚54132.74072265625& disk.nvme0n1.out‚54132.74072265625 disk.sda1.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 82.03515625 network.sent‚ 3.41815e+06) disk./.usagePercent‚78.94434366508743 disk.dm-0.in‚ 6535.20703125 disk.sda.in‚0 disk.drbd1260.in‚0# memory_percent‚99.9427080154419 cpu‚25.329393043309874 disk.sda2.in‚0# disk.dm-0.out‚54132.74072265625 disk.sda3.in‚0 disk.sda3.out‚ 82.03515625 disk.drbd1260.out‚ 0.39453125& proc.memory.availableMB‚ 9.38671875* proc.memory.percent‚17.804741859436035 proc.cpu.threads‚17 network.recv‚932872# disk.nvme0c0n1.in‚ 6535.20703125! disk.nvme0n1.in‚ 6535.20703125 disk.sda1.in‚0›Õ‰#ü‚(:„ ùèêÔµã×ö gpu.0.gpu‚26.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚134.83' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.966 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Öƶ ký‚(:à ùèêÔ„£Ôù disk.dm-0.in‚ 6710.80078125# disk.dm-0.out‚54598.89404296875 disk.sda.out‚82.5625 disk.drbd1260.in‚0$ memory_percent‚99.94275569915771& proc.memory.availableMB‚ 9.37890625 cpu‚25.329033407321038( disk./.usagePercent‚78.9611213748514 disk.sda3.out‚82.5625 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.3984375# proc.memory.rssMB‚ 2917.12890625* proc.memory.percent‚17.804741859436035 proc.cpu.threads‚17 network.sent‚ 3.424142e+06 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 network.recv‚937489% disk./.usageGB‚1176.5777702331543 disk.sda.in‚0# disk.nvme0c0n1.in‚ 6710.80078125( disk.nvme0c0n1.out‚54598.89404296875! disk.nvme0n1.in‚ 6710.80078125& disk.nvme0n1.out‚54598.89404296875 disk.dm-1.out‚82.5625‚ðæ2þ- _wandbruntime‚2880  _runtime‚28802€ÇÿÁ & lr-AdamW/pg1‚0.0009151787080587685 & lr-AdamW/pg2‚0.0009151787080587685 trainer/global_step‚1649 % _timestamp‚1.7885236418512957e+09 _runtime‚2880.070956452 _step‚131R ¸Ì€Æ _step‚131 & lr-AdamW/pg1‚0.0009151787080587685 & lr-AdamW/pg2‚0.0009151787080587685 trainer/global_step‚1649 % _timestamp‚1.7885236418512957e+09 _runtime‚2880.070956452ƒab¡½2- _wandbruntime‚2880  _runtime‚2880»A´†È'‚ÂO TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚70003748447744 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚75865029 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11824733 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 320851968 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚66733832320000 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚70708464457216 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3270149795328 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3269916127744 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚66733850322944 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚66733832320000 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30392008 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3328295529472 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚44703380 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 trainer/global_step‚1649 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚44703380 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚66733850322944 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚44729722 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚70003748447744 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3270149795328 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚74036736046592 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3269916127744 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚74037056898560 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30392719 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18567984 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11824024 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 % _timestamp‚1.7885236418525949e+09 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚44729722 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 296911872 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3260879931273 _runtime‚2880.072622112 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚31135305 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3328592441344 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚63548936618256 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚31135305 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3328592441344 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚63548936618256 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚75865029 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11824733 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 320851968 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚66733832320000 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚70708464457216 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3270149795328 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3269916127744 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚66733850322944 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚66733832320000 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30392008 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3328295529472 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚44703380 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 trainer/global_step‚1649 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚44703380 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚66733850322944 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚44729722 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚70003748447744 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3270149795328„-ðjp2„- _wandbruntime‚2881  _runtime‚2881æ×VÑ&…ËM I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3272113834496 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30410081 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 308255744 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3271880153600 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚70044998510592 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11831975 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚75883652 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚31153580 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚74080420409856 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚66849076414863 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3271880153600 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚75910022 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚31153580 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11831295 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 $ _timestamp‚1.788523643739339e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18578786 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚70045250194432 _step‚133 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚66773136359936 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18578788 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3330580238336 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚63586484015040 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚66773118356992 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚31153582 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3272113834496 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚74080728665600 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚70750124487168 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 284315648 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚44756440 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚70750148427264 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚66849322435507 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3262838420467 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚75883652 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚63586466975680ƒ³“AÖ&†ÐM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31153582 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚44756440 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚44756440 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚70750148427264 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚66849322435507 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3262838420467 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚75883652 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚63586466975680 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3272113834496 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30410081 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 308255744 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3271880153600 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚70044998510592 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11831975 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚75883652 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚31153580 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚74080420409856 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚66849076414863 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3271880153600 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚75910022 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚31153580 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11831295 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 $ _timestamp‚1.788523643739339e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18578786 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚70045250194432 _step‚133 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚66773136359936 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18578788 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3330580238336 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚63586484015040 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚66773118356992 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚31153582 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3272113834496 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚74080728665600 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚70750124487168 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 284315648 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚68784289500597 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚46041332 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚72066266003968 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3368365501952 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚68784043480161 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚72792639019520 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚691 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3358817971781 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚68698152172544 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚72066517674496 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3428332253696 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚65425471528816 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104yòxÏÌ'™ÆO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚68698134169600 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚689 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚72066517674496 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31280886 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚32049055 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚68784289500597 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚46041332 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚72066266003968 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3368365501952 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚68784043480161 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚72792639019520 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚691 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3358817971781 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚68698152172544 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚72066517674496 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3428332253696 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚65425471528816 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚78090387 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚78116731 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12173491 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32049057 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _runtime‚2966.734153848 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19108086 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 294814720 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚65425454489456 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3368131834368 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚46041332 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚76220995213312 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚72792662959616 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31281577 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273ˆ?ÕÆ2š- _wandbruntime‚2968  _runtime‚2968úîûÒ ›ËM EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚32067330 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19118886 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31298844 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚72107767750656 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚46094392 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 312450048 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 trainer/global_step‚1699 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚72107767750656 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32067332 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12179958 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚72834346929664 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32067332 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚68737438209536 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚68737420206592 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3370095860224 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚78161724 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3370329541120 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12180659 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚68823532326571 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3370329541120 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32067330 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚46068024 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚68823778347215 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 $ _timestamp‚1.788523730043902e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚701 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3430321617408 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚78135354 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚68737438209536 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚68737420206592 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19118888 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3360776460975 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚46094392 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309¼ Ö&œÐM VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12179958 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚72834346929664 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32067332 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚68737438209536 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚68737420206592 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3370095860224 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚78161724 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3370329541120 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12180659 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚68823532326571 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3370329541120 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32067330 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚46068024 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚68823778347215 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 $ _timestamp‚1.788523730043902e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚701 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3430321617408 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚78135354 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚68737438209536 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚68737420206592 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19118888 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3360776460975 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚46094392 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚32067330 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19118886 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31298844 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚72107767750656 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚46094392 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 312450048 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 trainer/global_step‚1699 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚72107767750656 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32067332 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0‰Ä¬2- _wandbruntime‚2968  _runtime‚2968bÅî4UžÏ 1loss/SRqqt3/detection_loss‚1.0105333271125951 0loss/SRqqt4/detection_loss‚2.850808058914385 .loss/FBt1/detection_loss‚2.351950608632144 trainer/global_step‚1699 2loss/SRqqt2/assignment_loss‚1.7673774643948204 .loss/FRt2/detection_loss‚3.504985166920556 1loss/SRqqt1/detection_loss‚1.9609009264371333 &loss/total_loss‚52.216940663104545 /loss/FRt4/assignment_loss‚3.676528286933899 1loss/SRqqt3/assignment_loss‚1.313663862568016 0loss/FBt2/assignment_loss‚0.3127861548120349 1loss/FBt3/assignment_loss‚0.18394539784640074 /loss/FRt3/detection_loss‚1.5852034669624615 0loss/SRqqt2/detection_loss‚2.975692366298876 epoch‚1 _runtime‚ 2968.27407795 1loss/FBt4/assignment_loss‚0.27083423819761776 % _timestamp‚1.7885237300531015e+09 _step‚138 2loss/SRqqt1/assignment_loss‚2.1659638304740954 /loss/FBt3/detection_loss‚3.8168153762817383 .loss/FRt1/assignment_loss‚4.51792797115114 1loss/SRqqt4/assignment_loss‚1.781310722231865 1loss/FBt1/assignment_loss‚0.36190804114666847 .loss/FRt4/detection_loss‚2.623782058432698 .loss/FRt1/detection_loss‚2.488840969072448 .loss/FBt2/detection_loss‚2.684213862699621 /loss/FBt4/detection_loss‚1.1068842081447778 /loss/FRt2/assignment_loss‚5.281597521569994 0loss/FRt3/assignment_loss‚1.6224867758685595‹0ÀZŸÔ 1loss/SRqqt1/detection_loss‚1.9609009264371333 &loss/total_loss‚52.216940663104545 /loss/FRt4/assignment_loss‚3.676528286933899 1loss/SRqqt3/assignment_loss‚1.313663862568016 0loss/FBt2/assignment_loss‚0.3127861548120349 1loss/FBt3/assignment_loss‚0.18394539784640074 /loss/FRt3/detection_loss‚1.5852034669624615 0loss/SRqqt2/detection_loss‚2.975692366298876 epoch‚1 _runtime‚ 2968.27407795 1loss/FBt4/assignment_loss‚0.27083423819761776 % _timestamp‚1.7885237300531015e+09 _step‚138 2loss/SRqqt1/assignment_loss‚2.1659638304740954 /loss/FBt3/detection_loss‚3.8168153762817383 .loss/FRt1/assignment_loss‚4.51792797115114 1loss/SRqqt4/assignment_loss‚1.781310722231865 1loss/FBt1/assignment_loss‚0.36190804114666847 .loss/FRt4/detection_loss‚2.623782058432698 .loss/FRt1/detection_loss‚2.488840969072448 .loss/FBt2/detection_loss‚2.684213862699621 /loss/FBt4/detection_loss‚1.1068842081447778 /loss/FRt2/assignment_loss‚5.281597521569994 0loss/FRt3/assignment_loss‚1.6224867758685595 1loss/SRqqt3/detection_loss‚1.0105333271125951 0loss/SRqqt4/detection_loss‚2.850808058914385 .loss/FBt1/detection_loss‚2.351950608632144 trainer/global_step‚1699 2loss/SRqqt2/assignment_loss‚1.7673774643948204 .loss/FRt2/detection_loss‚3.504985166920556Š'*X ‚(:… ÓéêÔ‹ÆŸ÷ gpu.0.gpu‚21.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚102.715' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.543 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Õ˜x£r¡‚(:ç ÓéêÔ·ôŸø network.sent‚ 3.559795e+06% disk./.usageGB‚1177.0787200927734 disk.sda2.in‚0 disk.dm-0.in‚ 6870.13671875# disk.dm-0.out‚56300.60595703125$ memory_percent‚99.94492530822754# proc.memory.rssMB‚ 2917.12890625 disk.sda3.out‚ 84.921875! disk.nvme0n1.in‚ 6870.13671875& disk.nvme0n1.out‚56300.60595703125 disk.sda1.in‚0 disk.dm-1.in‚0% proc.memory.availableMB‚ 9.0234375 proc.cpu.threads‚17) disk./.usagePercent‚78.99474054025536 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda3.in‚0( disk.nvme0c0n1.out‚56300.60595703125 disk.dm-1.out‚ 84.921875* proc.memory.percent‚17.804741859436035 cpu‚25.327282619339833 network.recv‚963873 disk.drbd1260.out‚ 0.41015625 disk.sda2.out‚0 disk.sda.out‚ 84.921875# disk.nvme0c0n1.in‚ 6870.13671875 disk.sda1.out‚0ùY]Ÿ¢‚(:… âéêÔàÆŠ÷ gpu.0.gpu‚5.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚107.806' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.5612 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0®5üßm£‚(:â âéêÔ‰˜ðú% disk./.usageGB‚1177.0787200927734 disk.sda.out‚85.1875& disk.nvme0n1.out‚56555.32080078125$ memory_percent‚99.94490146636963# proc.memory.rssMB‚ 2917.12890625 cpu‚25.327019980710055 proc.cpu.threads‚17 disk.sda2.out‚0# disk.dm-0.out‚56555.32080078125 disk.sda3.in‚0 disk.sda3.out‚85.1875 disk.sda1.in‚0 network.recv‚968927 disk.drbd1260.in‚0 disk.sda2.in‚0( disk.nvme0c0n1.out‚56555.32080078125! disk.nvme0n1.in‚ 6870.13671875 disk.dm-1.in‚0& proc.memory.availableMB‚ 9.02734375* proc.memory.percent‚17.804741859436035 network.sent‚ 3.667429e+06) disk./.usagePercent‚78.99474054025536 disk.drbd1260.out‚ 0.41015625 disk.dm-0.in‚ 6870.13671875 disk.sda.in‚0# disk.nvme0c0n1.in‚ 6870.13671875 disk.sda1.out‚0 disk.dm-1.out‚85.1875ð¦+µm¤‚(:â ñéêÔ²¾›ø# disk.dm-0.out‚56872.67724609375 disk.sda.in‚0( disk.nvme0c0n1.out‚56872.67724609375& proc.memory.availableMB‚ 9.01953125# proc.memory.rssMB‚ 2917.12890625 cpu‚25.32693316651747 network.sent‚ 3.673271e+06( disk./.usagePercent‚78.9931655819047 disk.sda.out‚ 85.60546875 disk.sda3.out‚ 85.60546875 disk.nvme0c0n1.in‚ 7000.28125 disk.nvme0n1.in‚ 7000.28125 disk.sda1.out‚0 disk.dm-1.out‚ 85.60546875 network.recv‚973239 disk.sda2.in‚0 disk.sda2.out‚0& disk.nvme0n1.out‚56872.67724609375 disk.sda1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.4140625$ memory_percent‚99.94494915008545% disk./.usageGB‚1177.0552520751953 disk.dm-0.in‚ 7000.28125 disk.sda3.in‚0 disk.dm-1.in‚0* proc.memory.percent‚17.804741859436035 proc.cpu.threads‚17Á“ýý‘¥‚(:† ñéêÔŠ·Ãø gpu.0.gpu‚33.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚115.966' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.1932 gpu.0.smClock‚1290 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¹¢Xœ¦‚(:‘ €êêÔ¯øþ÷ gpu.0.gpu‚33.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚134.591' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.918200000000002 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0áb‹l§‚(:á €êêÔÎŽÖú disk.dm-1.out‚ 86.21484375 disk.sda.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 86.21484375) disk./.usagePercent‚78.99474054025536 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda.out‚ 86.21484375$ proc.memory.availableMB‚9.265625 cpu‚25.32675962630687 disk.nvme0c0n1.in‚ 7000.28125 disk.nvme0n1.in‚ 7000.28125 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 7000.28125# disk.dm-0.out‚57267.41162109375$ memory_percent‚99.94344711303711 network.recv‚977634% disk./.usageGB‚1177.0787200927734& disk.nvme0n1.out‚57267.41162109375 disk.sda1.in‚0 disk.drbd1260.out‚ 0.4140625# proc.memory.rssMB‚ 2917.12890625* proc.memory.percent‚17.804741859436035 proc.cpu.threads‚17 network.sent‚ 3.678637e+06( disk.nvme0c0n1.out‚57267.41162109375¼_`dj¨‚(:ß êêÔÌëýø disk.drbd1260.out‚ 0.41796875 disk.sda2.in‚0& disk.nvme0n1.out‚57564.58740234375# disk.dm-0.out‚57564.58740234375 disk.sda.in‚0 disk.sda3.out‚ 86.5078125$ memory_percent‚99.94344711303711# proc.memory.rssMB‚ 2917.12890625* proc.memory.percent‚17.804741859436035 cpu‚25.326982475987492 network.recv‚981071 disk.sda2.out‚0$ proc.memory.availableMB‚9.265625 proc.cpu.threads‚17 network.sent‚ 3.683521e+06% disk./.usageGB‚1177.0552520751953 disk.sda.out‚ 86.5078125 disk.nvme0c0n1.in‚ 7000.28125 disk.nvme0n1.in‚ 7000.28125 disk.sda1.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 86.5078125( disk./.usagePercent‚78.9931655819047 disk.dm-0.in‚ 7000.28125 disk.sda3.in‚0( disk.nvme0c0n1.out‚57564.58740234375 disk.sda1.in‚0 disk.drbd1260.in‚0óÀGš‘©‚(:† êêÔøä—ù gpu.0.gpu‚23.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚105.487' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.0974 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£m|Opª‚(:å žêêÔÑÌú÷) disk./.usagePercent‚78.99474054025536% disk./.usageGB‚1177.0787200927734 disk.sda2.out‚0 disk.dm-0.in‚ 7000.28125( disk.nvme0c0n1.out‚57815.06787109375 proc.cpu.threads‚17 disk.nvme0n1.in‚ 7000.28125& disk.nvme0n1.out‚57815.06787109375 disk.drbd1260.in‚0# disk.dm-0.out‚57815.06787109375 disk.sda.out‚ 87.12890625 disk.sda3.out‚ 87.12890625$ memory_percent‚99.94347095489502& proc.memory.availableMB‚ 9.26171875 network.sent‚ 3.688678e+06 disk.sda1.in‚0 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.41796875 disk.sda.in‚0# proc.memory.rssMB‚ 2917.12890625* proc.memory.percent‚17.804741859436035 cpu‚25.326303976595913 disk.sda1.out‚0 disk.dm-1.out‚ 87.12890625 disk.sda2.in‚0 disk.sda3.in‚0 disk.nvme0c0n1.in‚ 7000.28125 network.recv‚984818ïòƒ‚œ«‚(:‘ žêêÔ¨í¯ø gpu.0.gpu‚17.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚140.663' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.132600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¯CÍ2¬- _wandbruntime‚3056  _runtime‚3056ú…ÿÊÆ­À $ _timestamp‚1.788523818636499e+09 _runtime‚3056.857157964 _step‚139 & lr-AdamW/pg1‚0.0009133083586493448 & lr-AdamW/pg2‚0.0009133083586493448 trainer/global_step‚1749’ŸŒË®Å & lr-AdamW/pg1‚0.0009133083586493448 & lr-AdamW/pg2‚0.0009133083586493448 trainer/global_step‚1749 $ _timestamp‚1.788523818636499e+09 _runtime‚3056.857157964 _step‚139‹ÛmÚ‡2¯- _wandbruntime‚3056  _runtime‚3056ŒÈ²—¯'°ÂO [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚70662436019200 _step‚140 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3466347540992 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12520628 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚78405030050304 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3466581208576 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3456756012289 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚47379284 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚70662454022144 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚74128783560192 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3528168588288 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚684 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚32168814 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚70758745412305 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚32962805 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚74129035230720 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3466581208576 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19648184 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚80342089 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292717568 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚70662436019200 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚74876837521920 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚70758499391869 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚682 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3456527031213 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3527875870720 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 _runtime‚3056.860351043 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚47405626 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.aå‹¥Ÿllocated‚70662454022144=úujÍ'±ÇO ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3466347540992 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚32168130 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚74129035230720 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚80368433 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚47405626 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32962807 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19648186 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚74876861462016 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 % _timestamp‚1.7885238186390252e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 trainer/global_step‚1749 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚47379284 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚74128783560192 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚74876837521920 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚70758499391869 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚682 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3456527031213 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3527875870720 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 _runtime‚3056.860351043 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚47405626 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚70662454022144 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚70662436019200 _step‚140 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3466347540992 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12520628 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚78405030050304 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3466581208576 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3456756012289 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚47379284 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚70662454022144 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚74128783560192 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3528168588288 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚684 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚32168814 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚70758745412305 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚32962805 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚74129035230720 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3466581208576 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19648184 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚80342089 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292717568 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚70662436019200 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32981082 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19658988 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚78448702996992 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚32981080 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 284315648 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32186104 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚70701740059136 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚80387056 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19658986 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚634 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3468311566848 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚80413426 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚70701722056192 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚80413426 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 _step‚141 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚47405976 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚74170033623040 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32185468 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 $ _timestamp‚1.788523820936609e+09 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚70701722056192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚78448394741248 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚67339502718080 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 trainer/global_step‚1749 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚80387056 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚74918521491968 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3458485520199 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚74170285306880 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚74170285306880 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 308255744 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32981082 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12526482 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3468311566848 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚70797988238279 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚74918545432064 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3458714501483 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0«J|ðÖ&´ÐM J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚80387056 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19658986 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚634 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3468311566848 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚80413426 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚70701722056192 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚80413426 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 _step‚141 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚47405976 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚74170033623040 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32185468 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 $ _timestamp‚1.788523820936609e+09 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚70701722056192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚78448394741248 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚67339502718080 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 trainer/global_step‚1749 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚80387056 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚74918521491968 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3458485520199 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚74170285306880 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚74170285306880 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 308255744 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32981082 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12526482 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3468311566848 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚70797988238279 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚74918545432064 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3458714501483 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32981082 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19658988 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚78448702996992 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚32981080 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 284315648 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32186104 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚70701740059136 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0oéc—2µ- _wandbruntime‚3059  _runtime‚3059Ï¢ˆV¶Ð 2loss/SRqqt3/assignment_loss‚1.2595873705020784 /loss/FRt4/detection_loss‚1.9273831658768203 1loss/SRqqt3/detection_loss‚1.1567692481047285 0loss/FBt3/assignment_loss‚0.7584184511963811 /loss/FRt3/detection_loss‚1.9218966758857339 /loss/SRqqt4/detection_loss‚2.80902842245996 /loss/FBt4/detection_loss‚1.1080195468304486 /loss/FRt1/assignment_loss‚4.542557064443827 0loss/SRqqt1/detection_loss‚2.034650204703212 /loss/FBt3/detection_loss‚2.6969992177827016 1loss/SRqqt4/assignment_loss‚2.398377390578389 %loss/total_loss‚51.78682708621419 trainer/global_step‚1749 _step‚142 0loss/FRt3/assignment_loss‚2.0919197312856124 /loss/FBt2/detection_loss‚2.3415818866342306 /loss/FRt2/assignment_loss‚5.074750141663984 .loss/FRt4/assignment_loss‚3.44963046514763 2loss/SRqqt1/assignment_loss‚1.7138582011684775 1loss/FBt4/assignment_loss‚0.19528470075749346 .loss/FRt1/detection_loss‚2.528649851679802 0loss/FBt1/assignment_loss‚0.4129508734753332 0loss/FBt2/assignment_loss‚0.3736176875715955 /loss/FRt2/detection_loss‚3.2786041010509837 0loss/SRqqt2/detection_loss‚2.581060992819922 epoch‚1 % _timestamp‚1.7885238209515173e+09 _runtime‚3059.172053284 2loss/SRqqt2/assignment_loss‚2.3369366036994115 /loss/FBt1/detection_loss‚2.7942950908954325 kJ<·Õ _step‚142 0loss/FRt3/assignment_loss‚2.0919197312856124 /loss/FBt2/detection_loss‚2.3415818866342306 /loss/FRt2/assignment_loss‚5.074750141663984 .loss/FRt4/assignment_loss‚3.44963046514763 2loss/SRqqt1/assignment_loss‚1.7138582011684775 1loss/FBt4/assignment_loss‚0.19528470075749346 .loss/FRt1/detection_loss‚2.528649851679802 0loss/FBt1/assignment_loss‚0.4129508734753332 0loss/FBt2/assignment_loss‚0.3736176875715955 /loss/FRt2/detection_loss‚3.2786041010509837 0loss/SRqqt2/detection_loss‚2.581060992819922 epoch‚1 % _timestamp‚1.7885238209515173e+09 _runtime‚3059.172053284 2loss/SRqqt2/assignment_loss‚2.3369366036994115 /loss/FBt1/detection_loss‚2.7942950908954325 2loss/SRqqt3/assignment_loss‚1.2595873705020784 /loss/FRt4/detection_loss‚1.9273831658768203 1loss/SRqqt3/detection_loss‚1.1567692481047285 0loss/FBt3/assignment_loss‚0.7584184511963811 /loss/FRt3/detection_loss‚1.9218966758857339 /loss/SRqqt4/detection_loss‚2.80902842245996 /loss/FBt4/detection_loss‚1.1080195468304486 /loss/FRt1/assignment_loss‚4.542557064443827 0lo™úù"Úss/SRqqt1/detection_loss‚2.034650204703212 /loss/FBt3/detection_loss‚2.6969992177827016 1loss/SRqqt4/assignment_loss‚2.398377390578389 %loss/total_loss‚51.78682708621419 trainer/global_step‚1749Ž¦¥}^‘¸‚(:† ­êêÔθ÷ gpu.0.gpu‚20.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚101.896' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.3792 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ûräùs¹‚(:è ­êêÔÒ€°û disk.drbd1260.out‚0.421875) disk./.usagePercent‚78.99474054025536 disk.dm-0.in‚ 7153.83203125# disk.dm-0.out‚58280.67333984375 disk.sda.in‚0 disk.sda3.out‚ 87.5234375( disk.nvme0c0n1.out‚58280.67333984375& disk.nvme0n1.out‚58280.67333984375 disk.sda1.in‚0 disk.sda.out‚ 87.5234375 disk.sda1.out‚0 disk.dm-1.out‚ 87.5234375$ memory_percent‚99.94549751281738% proc.memory.availableMB‚ 8.9296875# proc.memory.rssMB‚ 2917.12890625* proc.memory.percent‚17.804741859436035 proc.cpu.threads‚17 disk.sda2.out‚0! disk.nvme0n1.in‚ 7153.83203125 disk.dm-1.in‚0 disk.drbd1260.in‚0 cpu‚25.325396242058172 network.sent‚ 3.694554e+06 network.recv‚988967% disk./.usageGB‚1177.0787200927734 disk.sda2.in‚0 disk.sda3.in‚0# disk.nvme0c0n1.in‚ 7153.83203125µª‘º‚(:† ¼êêԾѨ÷ gpu.0.gpu‚23.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚102.662' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.5324 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚09Øo»‚(:ä ¼êêÔ­ã±ø# disk.dm-0.out‚58339.02099609375 disk.sda.in‚0 disk.sda3.out‚ 88.015625$ memory_percent‚99.94394779205322 disk.sda2.out‚0 disk.sda1.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 88.015625 disk.drbd1260.out‚0.421875 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda.out‚ 88.015625 disk.sda3.in‚0# proc.memory.rssMB‚ 2917.12890625* proc.memory.percent‚17.804741859436035 cpu‚25.32436465174311 disk.dm-0.in‚ 7153.83203125# disk.nvme0c0n1.in‚ 7153.83203125( disk.nvme0c0n1.out‚58339.02099609375! disk.nvme0n1.in‚ 7153.83203125& disk.nvme0n1.out‚58339.02099609375 disk.sda1.in‚0 disk.drbd1260.in‚0& proc.memory.availableMB‚ 9.18359375 network.sent‚ 3.803383e+06 network.recv‚995882( disk./.usagePercent‚78.9931655819047% disk./.usageGB‚1177.0552520751953¶†!¼‚(:… ËêêÔλÅù gpu.0.gpu‚16.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.031' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.6062 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0sšVQv½‚(:ë ËêêÔ÷Ýñû disk.sda2.in‚0 disk.dm-0.in‚ 7153.83203125 disk.sda3.in‚0# proc.memory.rssMB‚ 2917.12890625 network.sent‚ 3.808337e+06# disk.nvme0c0n1.in‚ 7153.83203125( disk.nvme0c0n1.out‚58704.33837890625! disk.nvme0n1.in‚ 7153.83203125 disk.sda1.out‚0 disk.sda.in‚0 disk.sda.out‚ 88.37109375 disk.sda3.out‚ 88.37109375% disk./.usageGB‚1177.0552520751953 disk.sda2.out‚0# disk.dm-0.out‚58704.33837890625$ memory_percent‚99.94397163391113* proc.memory.percent‚17.804741859436035 disk.drbd1260.out‚ 0.4296875& disk.nvme0n1.out‚58704.33837890625 disk.sda1.in‚0 disk.drbd1260.in‚0% proc.memory.availableMB‚ 9.1796875 cpu‚25.323633345182763 proc.cpu.threads‚17 network.recv‚999246( disk./.usagePercent‚78.9931655819047 disk.dm-1.in‚0 disk.dm-1.out‚ 88.37109375i@_!¾‚(:„ ÚêêÔ³‚ö gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚97.961' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.5922 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ž³{¿‚(:ð ÚêêÔ‡˜ˆø cpu‚25.322791927949044( disk./.usagePercent‚78.9931655819047 disk.sda2.in‚0 disk.dm-0.in‚ 7153.83203125# disk.dm-0.out‚59079.13525390625( disk.nvme0c0n1.out‚59079.13525390625! disk.nvme0n1.in‚ 7153.83203125 disk.sda1.out‚0 disk.dm-1.out‚ 88.70703125 disk.sda.in‚0 disk.sda1.in‚0 disk.dm-1.in‚0 disk.drbd1260.in‚0$ memory_percent‚99.94401931762695$ proc.memory.availableMB‚9.171875 proc.cpu.threads‚17 network.sent‚ 3.813528e+06 disk.sda2.out‚0 disk.sda.out‚ 88.70703125 disk.sda3.in‚0 disk.sda3.out‚ 88.70703125 disk.drbd1260.out‚ 0.4296875# proc.memory.rssMB‚ 2917.12890625* proc.memory.percent‚17.804741859436035 network.recv‚ 1.002874e+06% disk./.usageGB‚1177.0552520751953# disk.nvme0c0n1.in‚ 7153.83203125& disk.nvme0n1.out‚59079.13525390625îÍô8À‚(:… éêêÔï®Ë÷ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚101.483' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.2966 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/wò~Á‚(:ó éêêÔíú disk.dm-1.out‚ 89.07421875# proc.memory.rssMB‚ 2917.12890625* proc.memory.percent‚17.804741859436035 cpu‚25.321770833238983 proc.cpu.threads‚17% disk./.usageGB‚1177.0787200927734# disk.dm-0.out‚59588.78369140625 disk.sda.in‚0( disk.nvme0c0n1.out‚59588.78369140625 disk.sda1.in‚0 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.43359375$ memory_percent‚99.94397163391113 network.sent‚ 3.819415e+06 network.recv‚ 1.007313e+06 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 89.07421875# disk.nvme0c0n1.in‚ 7215.81640625% proc.memory.availableMB‚ 9.1796875) disk./.usagePercent‚78.99474054025536 disk.dm-0.in‚ 7215.81640625! disk.nvme0n1.in‚ 7215.81640625 disk.drbd1260.in‚0 disk.sda.out‚ 89.07421875& disk.nvme0n1.out‚59588.78369140625 disk.sda1.out‚0Ö•ƒ8‚(:… øêêÔçˆúö gpu.0.gpu‚20.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚92.922' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.5844 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þÐîDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚72733201324013 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚72626755871744 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12868799 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚76191301116416 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚72626737868800 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3554465071721 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚48743578 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3627999669760 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚48717236 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3564796915200 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚33876557 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚76961036024320 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚72626755871744 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3554694052797 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20188286 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 _step‚144 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚82593791 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚33876557 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚48743578 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚76191552786944 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚82620135 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 _runtime‚3143.082067722 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚1799 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20188286 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 _step‚144 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚82593791 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚33876557 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚48743578 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚76191552786944 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚82620135 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 _runtime‚3143.082067722 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚1799 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚72733201324013 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚72626755871744 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12868799 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚76191301116416 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚72626737868800 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3554465071721 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚48743578 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3627999669760 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚48717236 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3564796915200 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚33876557 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚76961036024320 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚72626755871744s©Øv2Ê- _wandbruntime‚3143  _runtime‚3143î2œ¨«ËÊM SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12875057 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚82638758 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚76232551179264 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚76232802863104 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 $ _timestamp‚1.788523905777907e+09 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3629709862400 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚82665128 R?DeviceStatsMonitor.on_train_batch_end/req?ÏÑ¢%uested_bytes.all.freed‚72772444149987 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚72666023905792 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚72772690170631 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚69216037628640 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚80632429856768 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚69216020589280 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290607104 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3566527273472 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3566760954368 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3566527273472 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚33894830 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚48770296 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 trainer/global_step‚1799 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚72666041908736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚82638758 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚48743928 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚82665128 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚33074816 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚33894832 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚48743928 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚673 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12875728 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20199088 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3556423560707 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚76232802863104 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚671 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚72666023905792 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33074143 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 _step‚145 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚33894832 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚33894830 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚80632744403968 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314547200 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚72666041908736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20199086 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚77002743934464:\.Õ&ÌÏM VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚72666023905792 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚72772690170631 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚69216037628640 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚80632429856768 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚69216020589280 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290607104 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3566527273472 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3566760954368 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3566527273472 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚33894830 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚48770296 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 trainer/global_step‚1799 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚72666041908736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚82638758 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚48743928 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚82665128 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚33074816 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚33894832 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚48743928 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚673 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12875728 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20199088 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3556423560707 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚76232802863104 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚671 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚72666023905792 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33074143 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 _step‚145 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚33894832 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚33894830 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚80632744403968 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314547200 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚72666041908736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20199086 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚77002743934464 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12875057 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚82638758 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚76232551179264 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚76232802863104 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 $ _timestamp‚1.788523905777907e+09 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3629709862400 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚82665128 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚72772444149987‘¸[’2Í- _wandbruntime‚3144  _runtime‚3144²7XÎÒ 2loss/SRqqt3/assignment_loss‚1.0996222334342594 epoch‚1 /loss/FRt1/assignment_loss‚4.347545333530592 /loss/FRt4/assignment_loss‚2.955071438331993 .loss/FBt1/detection_loss‚2.095590212128379 .loss/FBt3/detection_loss‚2.553425749142965 _step‚146 .loss/FRt1/detection_loss‚2.535286383136459 1loss/SRqqt3/detection_loss‚0.9815701300732228 1loss/FBt4/assignment_loss‚0.28421890038863645 1loss/FBt2/assignment_loss‚0.27453935655785244 0loss/FBt3/assignment_loss‚0.7199051979308327 0loss/SRqqt1/detection_loss‚2.016729927311341 0loss/SRqqt4/detection_loss‚3.184864580631256 &loss/total_loss‚49.785835838599766 2loss/SRqqt2/assignment_loss‚1.8812664361561047 /loss/FRt4/detection_loss‚1.9454810990362752 1loss/SRqqt2/detection_loss‚3.7167280404006733 /loss/FBt2/detection_loss‚2.6536480374634266 % _timestamp‚1.7885239057819364e+09 1loss/FBt1/assignment_loss‚0.45682364220599747 .loss/FBt4/detection_loss‚0.924949522116291 2loss/SRqqt1/assignment_loss‚1.7374677835032344 .loss/FRt2/detection_loss‚2.922142336001763 trainer/global_step‚1799 /loss/FRt2/assignment_loss‚4.715021316821758 0loss/FRt3/assignment_loss‚1.9600001194260337 1loss/SRqqt4/assignment_loss‚2.167123195070487 /loss/FRt3/detection_loss‚1.6568148677999324 _runtime‚3144.001294827öÜŠ:]Ï× epoch‚1 /loss/FRt1/assignment_loss‚4.347545333530592 /loss/FRt4/assignment_loss‚2.955071438331993 .loss/FBt1/detection_loss‚2.095590212128379 .loss/FBt3/detection_loss‚2.553425749142965 _step‚146 .loss/FRt1/detection_loss‚2.535286383136459 1loss/SRqqt3/detection_loss‚0.9815701300732228 1loss/FBt4/assignment_loss‚0.28421890038863645 1loss/FBt2/assignment_loss‚0.27453935655785244 0loss/FBt3/assignment_loss‚0.7199051979308327 0loss/SRqqt1/detection_loss‚2.016729927311341 0loss/SRqqt4/detection_loss‚3.184864580631256 &loss/total_loss‚49.785835838599766 2loss/SRqqt2/assignment_loss‚1.8812664361561047 /loss/FRt4/detection_loss‚1.9454810990362752 1loss/SRqqt2/detection_loss‚3.7167280404006733 /loss/FBt2/detection_loss‚2.6536480374634266 % _timestamp‚1.7885239057819364e+09 1loss/FBt1/assignment_loss‚0.45682364220599747 .loss/FBt4/detection_loss‚0.924949522116291 2loss/SRqqt1/assignment_loss‚1.7374677835032344 .loss/FRt2/detection_loss‚2.922142336001763 trainer/global_step‚1799 /loss/FRt2/assignment_loss‚4.715021316821758 0loss/FRt3/assignment_loss‚1.9600001194260337 1loss/SRqqt4/assignment_loss‚2.167123195070487 /loss/FRt3/detection_loss‚1.6568148677999324 _runtime‚3144.001294827 2loss/SRqqt3/assignment_loss‚1.0996222334342594’{oÛЂ(:… ‡ëêÔɉð÷ gpu.0.gpu‚22.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.031' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.6062 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘íAÁqÑ‚(:æ ‡ëêÔØúƒù% proc.memory.availableMB‚ 8.9765625 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.dm-1.out‚ 89.984375 network.recv‚ 1.015414e+06( disk.nvme0c0n1.out‚59983.52001953125& disk.nvme0n1.out‚59983.52001953125 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.nvme0n1.in‚ 7317.8359375 disk.drbd1260.in‚0 disk.dm-0.in‚ 7317.8359375# disk.dm-0.out‚59983.52001953125 disk.sda3.out‚ 89.984375$ memory_percent‚99.94521141052246# proc.memory.rssMB‚ 2917.12890625* proc.memory.percent‚17.804741859436035 network.sent‚ 3.830716e+06) disk./.usagePercent‚78.99481836654577% disk./.usageGB‚1177.0798797607422 disk.sda1.in‚0 disk.sda3.in‚0 cpu‚25.319800361390158" disk.nvme0c0n1.in‚ 7317.8359375 disk.drbd1260.out‚0.4375 disk.sda.in‚0 disk.sda.out‚ 89.984375ÿí%‘Ò‚(:† –ëêÔ€Êõõ gpu.0.gpu‚24.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚101.967' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.3934 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0X6u´vÓ‚(:ë –ëêÔÙœù# proc.memory.rssMB‚ 2917.12890625 network.sent‚ 3.938177e+06% disk./.usageGB‚1177.0564155578613 disk.sda2.out‚0 disk.dm-0.in‚ 7317.8359375 disk.sda.out‚ 90.31640625( disk.nvme0c0n1.out‚60310.49755859375 disk.nvme0n1.in‚ 7317.8359375& disk.nvme0n1.out‚60310.49755859375) disk./.usagePercent‚78.99324366420267 disk.sda2.in‚0 disk.sda.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 90.31640625 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 90.31640625" disk.nvme0c0n1.in‚ 7317.8359375* proc.memory.percent‚17.804741859436035 cpu‚25.31877235121216 proc.cpu.threads‚17 network.recv‚ 1.02071e+06# disk.dm-0.out‚60310.49755859375 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚0.4375$ memory_percent‚99.94680881500244& proc.memory.availableMB‚ 8.714843753É‘Ô‚(:† ¥ëêÔáŽæö gpu.0.gpu‚33.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚112.053' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.4106 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÀùLµqÕ‚(:æ ¥ëêÔýÁµ÷ disk.nvme0n1.in‚ 7317.8359375 disk.sda1.out‚0 disk.dm-1.out‚ 90.6953125 disk.sda2.in‚0 disk.sda3.in‚0* proc.memory.percent‚17.804741859436035" disk.nvme0c0n1.in‚ 7317.8359375 disk.sda1.in‚0 disk.dm-1.in‚0 disk.sda.out‚ 90.6953125$ memory_percent‚99.94828701019287 cpu‚25.3180182077074 network.sent‚ 3.944186e+06 network.recv‚ 1.025346e+06 disk.drbd1260.out‚ 0.44140625! disk.dm-0.out‚60652.115234375 disk.sda3.out‚ 90.6953125& proc.memory.availableMB‚ 8.47265625# proc.memory.rssMB‚ 2917.12890625 proc.cpu.threads‚17) disk./.usagePercent‚78.99324366420267% disk./.usageGB‚1177.0564155578613& disk.nvme0c0n1.out‚60652.115234375$ disk.nvme0n1.out‚60652.115234375 disk.drbd1260.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 7317.8359375 disk.sda.in‚0ôû9‘Ö‚(:† ´ëêԩίö gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚108.331' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.6662 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0†êQf|ׂ(:ñ ´ëêÔŒ±›ú disk.dm-0.in‚ 7455.51953125 disk.sda1.in‚0& proc.memory.availableMB‚ 8.95703125 cpu‚25.316811594344784 proc.cpu.threads‚17 network.recv‚ 1.030349e+06) disk./.usagePercent‚78.92398005779134% disk./.usageGB‚1176.0243377685547" disk.dm-0.out‚61151.5908203125 disk.sda3.out‚ 91.27734375' disk.nvme0c0n1.out‚61151.5908203125 disk.drbd1260.out‚ 0.44140625$ memory_percent‚99.94533061981201 network.sent‚ 3.950479e+06 disk.sda3.in‚0# disk.nvme0c0n1.in‚ 7455.51953125! disk.nvme0n1.in‚ 7455.51953125% disk.nvme0n1.out‚61151.5908203125 disk.sda1.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 91.27734375 disk.sda.in‚0 disk.sda.out‚ 91.27734375 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 2917.12890625* proc.memory.percent‚17.804741859436035?½­Ã2Ø- _wandbruntime‚3208  _runtime‚3208ŸééiÆÙÀ & lr-AdamW/pg1‚0.0009110596874680906 & lr-AdamW/pg2‚0.0009110596874680906 trainer/global_step‚1849 $ _timestamp‚1.788523970179625e+09 _runtime‚3208.399879357 _step‚147€m„ÚËÚÅ trainer/global_step‚1849 $ _timestamp‚1.788523970179625e+09 _runtime‚3208.399879357 _step‚147 & lr-AdamW/pg1‚0.0009110596874680906 & lr-AdamW/pg2‚0.0009110596874680906“KÊ4‰2Û- _wandbruntime‚3208  _runtime‚3208  +|•ÜÁO [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚74591039718400 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20728384 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚34790307 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3652632093305 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚50055188 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚681 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚71055008103056 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3663012621824 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13213492 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3727876383232 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚78254070343168 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3662778954240 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚50081530 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚79045258466816 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 trainer/global_step‚1849 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20728386 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34790305 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284328960 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚683 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 _runtime‚3208.401781512 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚78254070343168 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚74591057721344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 OIDeviceStatsMonitor.on_train_batch_staMÊÁ2rt/inactive_split.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚71055025142416 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚84871837 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚34790307 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34790305 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3652403112229 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚78253818672640 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚74591039718400 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3663012621824 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚74707657235721 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096ï²ùïÌ'ÝÆO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚681 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚71055008103056 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3663012621824 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13213492 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3727876383232 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚78254070343168 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3662778954240 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚50081530 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚79045258466816 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 trainer/global_step‚1849 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20728386 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34790305 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284328960 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚683 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 _runtime‚3208.401781512 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚78254070343168 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚74591057721344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚71055025142416 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚84871837 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚34790307 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34790305 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3652403112229 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚78253818672640 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚74591039718400 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3663012621824 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚74707657235721 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚74591039718400 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20728384 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚34790307 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3652632093305 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚50055188 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2”XÿD2Þ- _wandbruntime‚3209  _runtime‚3209h£MÑ&ßËM J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚50081880 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 308255744 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3729590000128 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚74630343758336 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚82816508496896 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚74746900061695 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚50108248 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚34808580 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚33960010 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚34808582 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚79086918496768 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 _step‚149 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20739186 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3664742980096 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3664976660992 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚84890460 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚78295068735488 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚74630325755392 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚688 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13220822 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚74747146082339 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33959320 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3654590582499 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚71092555499840 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚84916830 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3664742980096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 284315648 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34808582 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚78295320419328 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13220134 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34808580 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚78295320419328 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚71092538460480 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚84916830 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0V##>Ö&àÐM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3664976660992 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚84890460 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚78295068735488 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚74630325755392 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚688 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13220822 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚74747146082339 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33959320 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3654590582499 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚71092555499840 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚84916830 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3664742980096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 284315648 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34808582 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚78295320419328 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13220134 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34808580 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚78295320419328 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚71092538460480 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚84916830 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚50081880 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 308255744 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3729590000128 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚74630343758336 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚82816508496896 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚74746900061695 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚50108248 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚34808580 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚33960010 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚34808582 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚79086918496768 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 _step‚149 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20739186 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3664742980096 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216•$–Ã2á- _wandbruntime‚3209  _runtime‚32094õÕŠÆâÓ 1loss/FBt4/assignment_loss‚0.18712982724570223 1loss/SRqqt3/detection_loss‚1.0626004257860282 trainer/global_step‚1849 epoch‚1 0loss/FRt1/assignment_loss‚4.4247869268844005 2loss/SRqqt3/assignment_loss‚1.1339554379228503 1loss/SRqqt1/detection_loss‚2.4678060537890385 0loss/FRt3/assignment_loss‚2.1443349100076237 _step‚150 0loss/FBt1/assignment_loss‚0.5615814113512329 /loss/FRt3/detection_loss‚1.54@àúz“74350097087712 /loss/FRt4/detection_loss‚2.0378691921833645 .loss/FBt3/detection_loss‚2.292269309361776 1loss/SRqqt1/assignment_loss‚1.591779655825935 .loss/FRt2/detection_loss‚2.903987303376198 0loss/SRqqt2/detection_loss‚3.292579467807497 /loss/FBt1/detection_loss‚2.0785527378320694 .loss/FBt2/detection_loss‚2.621033407174624 &loss/total_loss‚49.171829704759126 0loss/FBt3/assignment_loss‚0.1110792828258127 .loss/FRt1/detection_loss‚2.747271635030445 /loss/FRt2/assignment_loss‚4.609501438481467 2loss/SRqqt4/assignment_loss‚1.9332193166017533 1loss/FBt2/assignment_loss‚0.25843979809157575 0loss/SRqqt4/detection_loss‚2.658041164278984 % _timestamp‚1.7885239711257637e+09 _runtime‚3209.345737121 0loss/FRt4/assignment_loss‚3.3645358002463053 1loss/SRqqt2/assignment_loss‚2.271592210446085 /loss/FBt4/detection_loss‚0.8704479824995897dxB^ãØ .loss/FRt1/detection_loss‚2.747271635030445 /loss/FRt2/assignment_loss‚4.609501438481467 2loss/SRqqt4/assignment_loss‚1.9332193166017533 1loss/FBt2/assignment_loss‚0.25843979809157575 0loss/SRqqt4/detection_loss‚2.658041164278984 % _timestamp‚1.7885239711257637e+09 _runtime‚3209.345737121 0loss/FRt4/assignment_loss‚3.3645358002463053 1loss/SRqqt2/assignment_loss‚2.271592210446085 /loss/FBt4/detection_loss‚0.8704479824995897 1loss/FBt4/assignment_loss‚0.18712982724570223 1loss/SRqqt3/detection_loss‚1.0626004257860282 trainer/global_step‚1849 epoch‚1 0loss/FRt1/assignment_loss‚4.4247869268844005 2loss/SRqqt3/assignment_loss‚1.1339554379228503 1loss/SRqqt1/detection_loss‚2.4678060537890385 0loss/FRt3/assignment_loss‚2.1443349100076237 _step‚150 0loss/FBt1/assignment_loss‚0.5615814113512329 /loss/FRt3/detection_loss‚1.5474350097087712 /loss/FRt4/detection_loss‚2.0378691921833645 .loss/FBt3/detection_loss‚2.292269309361776 1loss/SRqqt1/assignment_loss‚1.591779655825935 .loss/FRt2/detection_loss‚2.903987303376198 0loss/SRqqt2/detection_loss‚3.292579467807497 /loss/FBt1/detection_loss‚2.0785527378320694 .loss/FBt2/detection_loss‚2.621033407174624 &loss/total_loss‚49.171829704759126 0loss/FBt3/assignment_loss‚0.1110792828258127–žóp›ä‚(: ÃëêÔ ¼÷ gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚33.0 gpu.0.powerWatts‚135.62' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.124000000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 c¬2qå‚(:æ ÃëêÔÓ”Öù disk.sda.out‚ 91.71484375 disk.sda3.in‚0# disk.nvme0c0n1.in‚ 7455.51953125 disk.sda1.in‚0 disk.sda3.out‚ 91.71484375! disk.nvme0n1.in‚ 7455.51953125" disk.nvme0n1.out‚ 61415.5703125* proc.memory.percent‚17.804741859436035 cpu‚25.315669454875568 network.sent‚ 3.955498e+06 disk.sda2.out‚0 disk.dm-0.out‚ 61415.5703125$ disk.nvme0c0n1.out‚ 61415.5703125# proc.memory.rssMB‚ 2917.12890625 proc.cpu.threads‚17 network.recv‚ 1.033578e+06) disk./.usagePercent‚78.92398184984408 disk.sda1.out‚0 disk.dm-1.out‚ 91.71484375 disk.drbd1260.in‚0 disk.dm-0.in‚ 7455.51953125# memory_percent‚99.9455213546753& proc.memory.availableMB‚ 8.92578125% disk./.usageGB‚1176.0243644714355 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.4453125 disk.sda2.in‚0 disk.sda.in‚0ÙǔМæ‚(:‘ ÒëêÔÅÞö gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚113.533' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.706599999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡$­uç‚(:ê ÒëêÔ¿–Âø disk.sda3.out‚ 92.02734375 disk.drbd1260.out‚ 0.4453125 disk.sda2.in‚0 disk.dm-0.out‚61415.91015625 cpu‚25.314624181430403# disk.nvme0c0n1.in‚ 7455.51953125 disk.drbd1260.in‚0 disk.sda2.out‚0& proc.memory.availableMB‚ 8.89453125* proc.memory.percent‚17.804741859436035 proc.cpu.threads‚17 network.recv‚ 1.038861e+06% disk./.usageGB‚1176.0009117126465 disk.sda.out‚ 92.02734375% disk.nvme0c0n1.out‚61415.91015625# disk.nvme0n1.out‚61415.91015625 disk.sda1.out‚0 disk.dm-1.in‚0$ memory_percent‚99.94571208953857 network.sent‚ 4.063201e+06) disk./.usagePercent‚78.92240791552358 disk.sda.in‚0! disk.nvme0n1.in‚ 7455.51953125 disk.sda1.in‚0 disk.dm-1.out‚ 92.02734375 disk.dm-0.in‚ 7455.51953125# proc.memory.rssMB‚ 2917.12890625 disk.sda3.in‚0›./‘è‚(:† áëêÔò­ž÷ gpu.0.gpu‚24.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚105.363' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.0726 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0y<œì{é‚(:ð áëêÔÃçÙù disk.drbd1260.out‚ 0.44921875 disk.sda.out‚ 92.4609375 disk.sda3.out‚ 92.4609375& proc.memory.availableMB‚ 8.90234375# proc.memory.rssMB‚ 2917.12890625 network.sent‚ 4.068989e+06( disk./.usagePercent‚78.9224043314181% disk./.usageGB‚1176.0008583068848& disk.nvme0n1.out‚61751.72021484375 disk.sda1.out‚0 disk.dm-1.in‚0 disk.sda2.in‚0 disk.sda.in‚0! disk.nvme0n1.in‚ 7455.51953125 disk.sda1.in‚0 disk.dm-1.out‚ 92.4609375 disk.sda3.in‚0$ memory_percent‚99.94566440582275* proc.memory.percent‚17.804741859436035 cpu‚25.313627157249265 proc.cpu.threads‚17( disk.nvme0c0n1.out‚61751.72021484375 disk.sda2.out‚0 disk.dm-0.in‚ 7455.51953125# disk.dm-0.out‚61751.72021484375 network.recv‚ 1.043322e+06# disk.nvme0c0n1.in‚ 7455.51953125 disk.drbd1260.in‚0yïôê‚(:… ðëêÔõú—ö gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚101.314' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.2628 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0×R¨xë‚(:í ðëêÔÜÌÖø% disk./.usageGB‚1176.0008583068848 disk.nvme0n1.in‚ 7607.046875 disk.sda1.in‚0 disk.dm-1.out‚ 92.97265625 disk.sda3.out‚ 92.97265625# proc.memory.rssMB‚ 2917.12890625 proc.cpu.threads‚17( disk./.usagePercent‚78.9224043314181 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda.out‚ 92.97265625& proc.memory.availableMB‚ 8.64453125* proc.memory.percent‚17.804741859436035 network.recv‚ 1.047819e+06! disk.nvme0c0n1.in‚ 7607.046875& disk.nvme0n1.out‚62088.52490234375 disk.drbd1260.out‚ 0.44921875 disk.dm-0.in‚ 7607.046875# disk.dm-0.out‚62088.52490234375 disk.sda.in‚0 disk.sda3.in‚0( disk.nvme0c0n1.out‚62088.52490234375 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0$ memory_percent‚99.94723796844482 cpu‚25.312976719966734 network.sent‚ 4.075187e+06}P\‘ì‚(:† ÿëêÔšû¯ö gpu.0.gpu‚19.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚130.302' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.0604 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Pžó}rí‚(:ç ÿëêÔõƒžù disk.nvme0n1.in‚ 7607.046875 network.recv‚ 1.051142e+06% disk./.usageGB‚1175.9019088745117 disk.dm-1.out‚ 93.45703125 disk.drbd1260.in‚0 disk.drbd1260.out‚0.453125 disk.sda2.in‚0$ memory_percent‚99.94726181030273$ proc.memory.availableMB‚8.640625) disk./.usagePercent‚78.91576375198743 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.dm-0.in‚ 7607.046875% disk.nvme0n1.out‚62497.3134765625# proc.memory.rssMB‚ 2917.12890625* proc.memory.percent‚17.804741859436035 network.sent‚ 4.080217e+06' disk.nvme0c0n1.out‚62497.3134765625 cpu‚25.312307080179583 proc.cpu.threads‚17 disk.dm-1.in‚0" disk.dm-0.out‚62497.3134765625 disk.sda.in‚0 disk.sda.out‚ 93.45703125 disk.sda3.in‚0 disk.sda3.out‚ 93.45703125! disk.nvme0c0n1.in‚ 7607.046875^*è2î- _wandbruntime‚3279  _runtime‚3279_š2‘ÇïÁ % _timestamp‚1.7885240414155521e+09 _runtime‚3279.635013687 _step‚151 & lr-AdamW/pg1‚0.0009097940546796961 & lr-AdamW/pg2‚0.0009097940546796961 trainer/global_step‚1899ïAá?ÌðÆ & lr-AdamW/pg1‚0.0009097940546796961 & lr-AdamW/pg2‚0.0009097940546796961 trainer/global_step‚1899 % _timestamp‚1.7885240414155521e+09 _runtime‚3279.635013687 _step‚151—ÆZÚ32ñ- _wandbruntime‚3279  _runtime‚3279‰Û3È'òÂO M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚34830934 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3827816354816 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚76681867126993 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13562450 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 320851968 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚81129433029120 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚1899 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚87123539 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚688 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚80316587899392 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 _runtime‚3279.636124639 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3760994660864 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21268486 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚84957273324032 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3760994660864 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚51393140 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚84956952472064 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚76555341568000 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚81129456969216 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚76682113147429 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚80316336228864 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚35704057 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚80316587899392 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚51393140 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3761228328448 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21268484 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚72931525974256 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚80316336228864 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚51419482 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚34831622 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13563136 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3750341152737 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 % _timestamp‚1.7885240414165719e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚35704057 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚72931543013616 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚51419482 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 296911872 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3761228328448 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚80316587899392 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚51393140 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3761228328448 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21268484 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚72931525974256 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚80316336228864 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚51419482 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚34831622 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13563136 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3750341152737 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 % _timestamp‚1.7885240414165719e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚35704057 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚72931543013616 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚51419482 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 296911872 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3761228328448 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚76681867126993 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13562450 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 320851968 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚81129433029120 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚1899 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚87123539 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚688 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚80316587899392 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 _runtime‚3279.636124639 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3760994660864 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21268486 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚84957273324032 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3760994660864 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚51393140 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚84956952472064 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚76555341568000 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚81129456969216 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚76682113147429˜ëœ¦2ô- _wandbruntime‚3280  _runtime‚3280âÃ.ã‘ õÌM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34848407 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚87168532 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚87168532 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 284315648 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3829519934464 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 _step‚153 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freeã¸qOAd‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚51446200 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚81171116999168 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚76594645607936 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3752528623007 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3762958686720 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚81171140939264 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3763192367616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚51419832 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 % _timestamp‚1.7885240423424509e+09 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚80357837975552 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚87142162 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚703 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3829804250112 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚72969073371040 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3762958686720 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21279286 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273Ña¿G×&öÑM OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚80357586291712 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚35722330 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚85000945189376 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3763192367616 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚701 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚87168532 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 284315648 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3829519934464 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 _step‚153 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚51446200 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚81171116999168 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚76594645607936 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3752528623007 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3762958686720 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚81171140939264 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3763192367616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚51419832 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 % _timestamp‚1.7885240423424509e+09 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚80357837975552 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚87142162 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚703 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3829804250112 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚72969073371040 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3762958686720 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21279286 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34848407 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚87168532 ODeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚82378853785088 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚35718077 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13909493 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21808584 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚52731092 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13910182 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚83213631531520 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3927614806528 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚691 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚89375241 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚89348897 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3859444035072 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚52757434 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚82379105455616 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35718768 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚78656569059137 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚78519643417600 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚82378853785088 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚689 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚78519661420544 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 _runtime‚3359.577455351 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚36617807 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚36617805 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚78656323038701 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 294814720 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21808586 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3848508174321 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3848279193245 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚78519643417600 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 trainer/global_step‚1949ÉÁ`ãË'‰ÅO SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚87141270278144 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3927319991808 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚156 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚82379105455616 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3859444035072 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚36617807 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚83213655471616 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚78656323038701 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 294814720 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21808586 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3848508174321 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3848279193245 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚78519643417600 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 trainer/global_step‚1949 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚78519661420544 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚82378853785088 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚35718077 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13909493 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21808584 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚52731092 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13910182 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚83213631531520 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3927614806528 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚691 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚89375241 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚89348897 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3859444035072 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚52757434 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚82379105455616 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35718768 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚78656569059137 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚78519643417600 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚82378853785088 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚689 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚78519661420544 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 _runtime‚3359.577455351 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚36617807 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚36617805 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚78558947457536 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚82420355531776 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36636080 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚74845574202880 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3861174393344 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚52784152 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚89420234 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚74845591242240 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35736235 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35735528 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚78695811885111 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚87184944994304 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3929308654080 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚89393864 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 320838656 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚87184624155648 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3861174393344 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3861408074240 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚82420355531776 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚52757784 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21819388 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚89420234 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3861174393344 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚52784152 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚89420234 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚74845591242240 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35736235 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35735528 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚78695811885111 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚87184944994304 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3929308654080 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚89393864 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 320838656 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚87184624155648 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3861174393344 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3861408074240 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚82420355531776 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚52757784 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.€ë; all.peak‚1319 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21819388 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚89420234 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚78558947457536 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚82420355531776 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36636080 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚74845574202880 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608eÅ(V2- _wandbruntime‚3360  _runtime‚3360Èd£’FÀ _runtime‚3360.928193943 /loss/FBt4/assignment_loss‚0.406281412112416 /loss/FRt3/assignment_loss‚2.028177499771118 0loss/FRt4/assignment_loss‚3.8701998378549303 2loss/SRqqt4/assignment_loss‚2.0970523711293936 1loss/SRqqt1/detection_loss‚2.2799135220773294 &loss/total_loss‚49.200659589503005 2loss/SRqqt2/assignment_loss‚1.7961760878562927 .loss/FRt1/detection_loss‚2.939823084643909 .loss/FRt4/detection_loss‚2.167139321016813 /loss/FRt1/assignment_loss‚3.626369535923004 0loss/FBt2/assignment_loss‚0.3650894958092168 1loss/SRqqt3/detection_loss‚0.9162624668870264 /loss/FRt2/assignment_loss‚5.247991065184276 1loss/FBt1/assignment_loss‚0.34441551902112777 /loss/FRt2/detection_loss‚3.1996274342139563 1loss/SRqqt4/detection_loss‚2.6782028699914613 /loss/FBt1/detection_loss‚2.7486630347039966 .loss/FBt4/detection_loss‚1.043675553140929 2loss/SRqqt3/assignment_loss‚1.1068710814770453 /loss/FBt2/detection_loss‚2.3255335092544556 /loss/FBt3/detection_loss‚1.6642459630966187 $ _timestamp‚1.788524122708475e+09 _step‚158 0loss/SRqqt1/assignment_loss‚1.76188460308494 loss/FBt3/assignment_loss‚0 1loss/SRqqt2/detection_loss‚2.8291781187057494 epoch‚1 trainer/global_step‚1949 /loss/FRt3/detection_loss‚1.7578862025469975+WK‘Å 2loss/SRqqt4/assignment_loss‚2.0970523711293936 1loss/SRqqt1/detection_loss‚2.2799135220773294 &loss/total_loss‚49.200659589503005 2loss/SRqqt2/assignment_loss‚1.7961760878562927 .loss/FRt1/detection_loss‚2.939823084643909 .loss/FRt4/detection_loss‚2.167139321016813 /loss/FRt1/assignment_loss‚3.626369535923004 0loss/FBt2/assignment_loss‚0.3650894958092168 1loss/SRqqt3/detection_loss‚0.9162624668870264 /loss/FRt2/assignment_loss‚5.247991065184276 1loss/FBt1/assignment_loss‚0.34441551902112777 /loss/FRt2/detection_loss‚3.1996274342139563 1loss/SRqqt4/detection_loss‚2.6782028699914613 /loss/FBt1/detection_loss‚2.7486630347039966 .loss/FBt4/detection_loss‚1.043675553140929 2loss/SRqqt3/assignment_loss‚1.1068710814770453 /loss/FBt2/detection_loss‚2.3255335092544556 /loss/FBt3/detection_loss‚1.6642459630966187 $ _timestamp‚1.788524122708475e+09 _step‚158 0loss/SRqqt1/assignment_loss‚1.76188460308494 loss/FBt3/assignment_loss‚0 1loss/SRqqt2/detection_loss‚2.8291781187057494 epoch‚1 trainer/global_step‚1949 /loss/FRt3/detection_loss‚1.7578862025469975 _runtime‚3360.928193943 /loss/FBt4/assignment_loss‚0.406281412112416 /loss/FRt3/assignment_loss‚2.028177499771118 0loss/FRt4/assignment_loss‚3.8701998378549303ž!>i*‘’‚(:† èìêÔ°‰‚ø gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚138.184' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.6368 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0RÜBw“‚(:ì èìêÔçÍ›ø disk.nvme0n1.in‚ 7929.5703125 disk.drbd1260.out‚ 0.46484375 disk.sda3.out‚ 96.6484375$ memory_percent‚99.94502067565918 cpu‚25.306764984339335 network.recv‚ 1.082886e+06 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0# disk.dm-0.out‚64230.22705078125 disk.sda.out‚ 96.6484375# proc.memory.rssMB‚ 2917.12890625 network.sent‚ 4.312382e+06$ disk./.usageGB‚1176.149024963379" disk.nvme0c0n1.in‚ 7929.5703125& disk.nvme0n1.out‚64230.22705078125 disk.dm-1.out‚ 96.6484375 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.dm-0.in‚ 7929.5703125) disk./.usagePercent‚78.93234792005552 disk.sda2.out‚0 disk.sda.in‚0 disk.sda3.in‚0% proc.memory.availableMB‚ 9.0078125* proc.memory.percent‚17.804741859436035 proc.cpu.threads‚17( disk.nvme0c0n1.out‚64230.22705078125B'°U‘”‚(:† ÷ìêÔÕÅÔö gpu.0.gpu‚24.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚110.446' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.0892 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0˜ŽAñy•‚(:î ÷ìêÔ¡äÿø network.sent‚ 4.409053e+06 network.recv‚ 1.087793e+06) disk./.usagePercent‚78.93654388354172% disk./.usageGB‚1176.2115478515625! disk.nvme0n1.in‚ 7946.62890625% disk.nvme0n1.out‚64527.5615234375 disk.sda1.out‚0 disk.dm-1.out‚ 97.17578125 disk.drbd1260.out‚0.46875 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 7946.62890625" disk.dm-0.out‚64527.5615234375* proc.memory.percent‚17.804741859436035 cpu‚25.306723408751704 disk.sda3.in‚0 disk.sda3.out‚ 97.17578125 disk.sda1.in‚0 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda.out‚ 97.17578125& proc.memory.availableMB‚ 9.00390625 proc.cpu.threads‚17# disk.nvme0c0n1.in‚ 7946.62890625' disk.nvme0c0n1.out‚64527.5615234375 disk.dm-1.in‚0$ memory_percent‚99.94504451751709# proc.memory.rssMB‚ 2917.12890625¼°âÑ–‚(:† †íêÔ¸þëö gpu.0.gpu‚33.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚112.053' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.4106 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0×äQJk—‚(:à †íêÔ¦’×÷ disk.drbd1260.out‚0.46875$ memory_percent‚99.94659423828125 proc.memory.availableMB‚8.75 disk.dm-0.in‚ 7946.62890625 disk.sda3.out‚ 97.515625# disk.nvme0c0n1.in‚ 7946.62890625! disk.nvme0n1.in‚ 7946.62890625% disk.nvme0n1.out‚64585.8154296875 disk.dm-1.out‚ 97.515625# proc.memory.rssMB‚ 2917.12890625* proc.memory.percent‚17.804741859436035 network.sent‚ 4.414291e+06 network.recv‚ 1.091348e+06 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 cpu‚25.30642485833473$ disk./.usageGB‚1176.336410522461' disk.nvme0c0n1.out‚64585.8154296875 disk.dm-1.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚78.94492352215248" disk.dm-0.out‚64585.8154296875 disk.sda.in‚0 disk.sda.out‚ 97.515625 disk.drbd1260.in‚0VšY›˜‚(: •íêÔäêö÷ gpu.0.gpu‚22.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚95.972' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚19.194399999999998 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0´Cdr™‚(:ç •íêÔÆëáù disk.drbd1260.in‚0 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda.out‚ 98.078125 disk.sda3.out‚ 98.078125! disk.nvme0n1.in‚ 7946.62890625$ disk.nvme0n1.out‚64855.083984375 disk.sda1.out‚0* proc.memory.percent‚17.804741859436035 cpu‚25.306077433663734 network.recv‚ 1.095828e+06 disk.sda2.out‚0! disk.dm-0.out‚64855.083984375# disk.nvme0c0n1.in‚ 7946.62890625& disk.nvme0c0n1.out‚64855.083984375 disk.dm-1.in‚0$ memory_percent‚99.94661808013916& proc.memory.availableMB‚ 8.74609375 disk.sda3.in‚0 disk.dm-1.out‚ 98.078125 disk.drbd1260.out‚ 0.47265625# proc.memory.rssMB‚ 2917.12890625 network.sent‚ 4.42014e+06) disk./.usagePercent‚78.94334856380183% disk./.usageGB‚1176.3129425048828 disk.dm-0.in‚ 7946.62890625 disk.sda.in‚0 disk.sda1.in‚0Ú²‡myš‚(:î ¤íêÔ¨‘¼ø network.sent‚ 4.426206e+06% disk./.usageGB‚1176.5846405029297# disk.nvme0c0n1.in‚ 7946.62890625 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda.in‚0 disk.sda3.in‚0 network.recv‚ 1.100223e+06) disk./.usagePercent‚78.96158244442056 disk.dm-1.out‚ 98.45703125 disk.drbd1260.in‚0! disk.dm-0.out‚65326.052734375 disk.sda3.out‚ 98.45703125& proc.memory.availableMB‚ 8.74609375# proc.memory.rssMB‚ 2917.12890625 disk.drbd1260.out‚ 0.47265625& disk.nvme0c0n1.out‚65326.052734375! disk.nvme0n1.in‚ 7946.62890625$ disk.nvme0n1.out‚65326.052734375* proc.memory.percent‚17.804741859436035 disk.dm-1.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 7946.62890625 disk.sda.out‚ 98.45703125$ memory_percent‚99.94661808013916 cpu‚25.305855856796196 proc.cpu.threads‚17ô2ÇŸ‘›‚(:† ¤íêÔÑŽÀø gpu.0.gpu‚13.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚102.254' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.4508 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0&9Ê2œ- _wandbruntime‚3443  _runtime‚3443éjƒcÇÁ & lr-AdamW/pg1‚0.0009069815134322868 & lr-AdamW/pg2‚0.0009069815134322868 trainer/global_step‚1999 % _timestamp‚1.7885242049288087e+09 _runtime‚3443.149414205 _step‚159õ!cÌžÆ % _timestamp‚1.7885242049288087e+09 _runtime‚3443.149414205 _step‚159 & lr-AdamW/pg1‚0.0009069815134322868 & lr-AdamW/pg2‚0.0009069815134322868 trainer/global_step‚1999ŸRNSN2Ÿ- _wandbruntime‚3443  _runtime‚3443¯Ä>FÈ' ÂO WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3957426074112 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚54069044 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4027372210688 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 % _timestamp‚1.7885242049327118e+09 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚687 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22348686 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚84441371341312 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 trainer/global_step‚1999 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36609453 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚89325226184704 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚80631024970845 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37531557 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚689 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚80630778950409 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚85297853974016 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 301106176 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚76684578756016 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚80483963270144 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚54069044 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚85297830033920 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚37531555 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚76684561716656 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3946217233753 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚91600599 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚80483945267200 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚84441623011840 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚89324901138432 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14260080 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 325046272 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚80483963270144 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚84441623011840 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚37531555 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22348684 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚84441371341312 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 trainer/global_step‚1999 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36609453 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚89325226184704 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚80631024970845 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37531557 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚689 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚80630778950409 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚85297853974016 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 301106176 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚76684578756016 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚80483963270144 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚54069044 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚85297830033920 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚37531555 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚76684561716656 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3946217233753 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚91600599 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚80483945267200 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚84441623011840 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚89324901138432 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14260080 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 325046272 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚80483963270144 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚84441623011840 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚37531555 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22348684 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚80523249307136 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3959623780864 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 _step‚161 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚84482873088000 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3948175722739 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36626391 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚1999 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37549832 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22359486 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚80670513817463 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3959623780864 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14267574 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚91645566 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3959390099968 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚54095736 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4029064016384 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3948404704023 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37549830 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37549830 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14266905 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚85339514003968 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚54122104 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚671 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚669 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 294801408 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚91645566 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22359488 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚76722109113440 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚76722092074080 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37549832 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚80523249307136 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚91671936 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4029358817792 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚80523231304192 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚85339537944064 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚80523231304192 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚36627062 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚84482873088000 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚80670267796819 % _timestamp‚1.7885242072171319e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚91671936 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚54122104 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚671 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚669 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 294801408 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚91645566 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22359488 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚76722109113440 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚76722092074080 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37549832 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚80523249307136 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚91671936 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4029358817792 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚80523231304192 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚85339537944064 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚80523231304192 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚36627062 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚84482873088000 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚80670267796819 % _timestamp‚1.7885242072171319e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚91671936 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚80523249307136 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3959623780864 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 _step‚161 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚84482873088000 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3948175722739 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36626391 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚1999 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37549832 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22359486 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚80670513817463 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3959623780864 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14267574 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚91645566 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3959390099968 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚54095736 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4029064016384 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3948404704023 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37549830 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37549830¡ !„52¥- _wandbruntime‚3445  _runtime‚3445F3íg\¦Ö .loss/FBt3/detection_loss‚3.211881071329117 trainer/global_step‚1999 % _timestamp‚1.7885242072221913e+09 0loss/FRt2/assignment_loss‚4.8186740168818725 0loss/FBt2/assignment_loss‚0.6129622035163187 /loss/FRt2/detection_loss‚3.0579905245039196 /loss/FBt1/detection_loss‚2.1878020659736963 2loss/SRqqt2/assignment_loss‚2.0005408951214383 2loss/SRqqt3/assignment_loss‚1.0771064281601597 2loss/SRqqt4/assignment_loss‚2.6063688567706516 1loss/FBt1/assignment_loss‚0.37450874482197316 1loss/SRqqt3/detection_loss‚1.0349150267947052 _step‚162 1loss/FBt3/assignment_loss‚0.09404343168716878 0loss/SRqqt1/detection_loss‚1.681306347846985 epoch‚1 0loss/FRt4/assignment_loss‚3.2672173157334328 /loss/FRt4/detection_loss‚1.8252779899963312 /loss/FBt2/detection_loss‚2.1523305320221446 %loss/total_loss‚49.72516916862936 2loss/SRqqt1/assignment_loss‚1.8223662748932838 .loss/FRt1/detection_loss‚2.818440143018961 1loss/SRqqt2/detection_loss‚3.1236095371700467 /loss/FBt4/detection_loss‚0.9120959122147825 _runtime‚3445.442129876 0loss/FRt3/assignment_loss‚1.8360548693625653 0loss/FBt4/assignment_loss‚0.2517547302626484 0loss/SRqqt4/detection_loss‚3.073989280632564 0loss/FRt1/assignment_loss‚3.9522273279726505 .loss/FRt3/detection_loss‚1.931705641941946†]³®a§Û 0loss/FRt3/assignment_loss‚1.8360548693625653 0loss/FBt4/assignment_loss‚0.2517547302626484 0loss/SRqqt4/detection_loss‚3.073989280632564 0loss/FRt1/assignment_loss‚3.9522273279726505 .loss/FRt3/detection_loss‚1.931705641941946 .loss/FBt3/detection_loss‚3.211881071329117 trainer/global_step‚1999 % _timestamp‚1.7885242072221913e+09 0loss/FRt2/assignment_loss‚4.8186740168818725 0loss/FBt2/assignment_loss‚0.6129622035163187 /loss/FRt2/detection_loss‚3.0579905245039196 /loss/FBt1/detection_loss‚2.1878020659736963 2loss/SRqqt2/assignment_loss‚2.0005408951214383 2loss/SRqqt3/assignment_loss‚1.0771064281601597 2loss/SRqqt4/assignment_loss‚2.6063688567706516 1loss/FBt1/assignment_loss‚0.37450874482197316 1loss/SRqqt3/detection_loss‚1.0349150267947052 _step‚162 1loss/FBt3/assignment_loss‚0.09404343168716878 0loss/SRqqt1/detection_loss‚1.681306347846985 epoch‚1 0loss/FRt4/assignment_loss‚3.2672173157334328 /loss/FRt4/detection_loss‚1.8252779899963312 /loss/FBt2/detection_loss‚2.1523305320221446 %loss/total_loss‚49.72516916862936 2loss/SRqqt1/assignment_loss‚1.8223662748932838 .loss/FRt1/detection_loss‚2.818440143018961 1loss/SRqqt2/detection_loss‚3.1236095371700467 /loss/FBt4/detection_loss‚0.9120959122147825 _runtime‚3445.442129876¢ŠôÍk¨‚(:à ³íêÔ´ðï÷ disk.sda.in‚0# memory_percent‚99.9495267868042 disk.dm-1.in‚0 disk.dm-1.out‚ 99.06640625! disk.dm-0.out‚65668.330078125 disk.sda.out‚ 99.06640625# proc.memory.rssMB‚ 2917.12890625 cpu‚25.305768242982097 proc.cpu.threads‚17 disk.sda3.in‚0& disk.nvme0c0n1.out‚65668.330078125 disk.nvme0n1.in‚ 8079.5625 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 8079.5625* proc.memory.percent‚17.804741859436035 network.recv‚ 1.103846e+06% disk./.usageGB‚1176.5631523132324 disk.sda3.out‚ 99.06640625$ disk.nvme0n1.out‚65668.330078125 disk.drbd1260.in‚0& proc.memory.availableMB‚ 8.26953125 network.sent‚ 4.431211e+06) disk./.usagePercent‚78.96014035398018 disk.nvme0c0n1.in‚ 8079.5625 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.out‚ 0.4765625lbcs›©‚(: ³íêÔ°ñä÷ gpu.0.gpu‚23.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚99.844' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚19.968799999999998 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0H•‘ª‚(:† ÂíêԞ܅÷ gpu.0.gpu‚19.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚101.728' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.3456 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŽQÍgf«‚(:Û ÂíêÔÒÑÊø disk.dm-0.in‚ 8079.5625 disk.sda.in‚0 disk.sda3.in‚0 disk.drbd1260.out‚ 0.4765625& proc.memory.availableMB‚ 7.98828125* proc.memory.percent‚17.804741859436035) disk./.usagePercent‚78.93489084289314 disk.dm-0.out‚66055.48046875 disk.sda3.out‚ 99.6484375 disk.nvme0n1.in‚ 8079.5625# disk.nvme0n1.out‚66055.48046875 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚99.95124340057373 network.recv‚ 1.109352e+06% disk./.usageGB‚1176.1869163513184 disk.sda2.in‚0 disk.nvme0c0n1.in‚ 8079.5625 disk.dm-1.out‚ 99.6484375 proc.cpu.threads‚17 network.sent‚ 4.538617e+06 disk.sda.out‚ 99.6484375 disk.dm-1.in‚0 disk.drbd1260.in‚0# proc.memory.rssMB‚ 2917.12890625 cpu‚25.305448843926115% disk.nvme0c0n1.out‚66055.48046875 disk.sda2.out‚0@H õo¬‚(:ä ÑíêÔã“Áø cpu‚25.305001659655353 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.48046875 disk.sda2.in‚0 proc.cpu.threads‚17% disk./.usageGB‚1174.9839401245117 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda3.in‚0 disk.nvme0c0n1.in‚ 8079.5625( disk.nvme0c0n1.out‚66478.38330078125 disk.nvme0n1.in‚ 8079.5625* proc.memory.percent‚17.804741859436035 network.sent‚ 4.544564e+06 network.recv‚ 1.11355e+06) disk./.usagePercent‚78.85415809894786 disk.sda2.out‚0 disk.dm-0.in‚ 8079.5625& disk.nvme0n1.out‚66478.38330078125$ memory_percent‚99.95288848876953# proc.memory.availableMB‚7.71875 disk.dm-1.out‚ 100.0234375# disk.dm-0.out‚66478.38330078125 disk.sda.out‚ 100.0234375 disk.sda3.out‚ 100.0234375 disk.sda1.in‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 2917.12890625úwV­‚(:„ ÑíêÔìâ¾ù gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.031' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.6062 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ýp¤Í®‚(:„ àíêԳɖö gpu.0.gpu‚6.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚102.015' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.403 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0E­üv¯‚(:ë àíêÔµ›†ø# proc.memory.rssMB‚ 2917.12890625 cpu‚25.304634908541352) disk./.usagePercent‚78.88771812661142 disk.nvme0n1.in‚ 8079.5625 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 proc.cpu.threads‚17 network.recv‚ 1.118527e+06( disk.nvme0c0n1.out‚66536.59423828125 disk.dm-1.in‚0 disk.dm-1.out‚ 100.39453125 disk.drbd1260.out‚ 0.48046875 disk.dm-0.in‚ 8079.5625 disk.sda.out‚ 100.39453125 disk.sda3.in‚0 disk.nvme0c0n1.in‚ 8079.5625& disk.nvme0n1.out‚66536.59423828125 disk.sda2.out‚0# disk.dm-0.out‚66536.59423828125 disk.sda.in‚0$ memory_percent‚99.95291233062744& proc.memory.availableMB‚ 7.71484375* proc.memory.percent‚17.804741859436035 network.sent‚ 4.550879e+06% disk./.usageGB‚1175.4840087890625 disk.drbd1260.in‚0 disk.sda3.out‚ 100.39453125án²ï‘°‚(:† ïíêÔ¹¤’÷ gpu.0.gpu‚24.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚103.843' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.7686 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 Uv±‚(:ë ïíêÔ·”•ø" disk.dm-0.out‚66854.0146484375% disk.nvme0n1.out‚66854.0146484375# proc.memory.rssMB‚ 2917.12890625* proc.memory.percent‚17.804741859436035% disk./.usageGB‚1175.4839515686035 disk.drbd1260.in‚0 disk.sda.out‚ 100.93359375 disk.sda3.in‚0" disk.nvme0c0n1.in‚ 8185.2421875 disk.sda1.in‚0 network.recv‚ 1.121601e+06 disk.dm-1.out‚ 100.93359375 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 8185.2421875 disk.sda3.out‚ 100.93359375' disk.nvme0c0n1.out‚66854.0146484375$ memory_percent‚99.95298385620117 network.sent‚ 4.555896e+06 disk.drbd1260.out‚0.484375 disk.sda.in‚0 disk.nvme0n1.in‚ 8185.2421875 disk.sda1.out‚0$ proc.memory.availableMB‚7.703125 cpu‚25.30433284298255 proc.cpu.threads‚17( disk./.usagePercent‚78.8877142864984 disk.dm-1.in‚0 £ü~2²- _wandbruntime‚3522  _runtime‚3522¶ôNdzÁ & lr-AdamW/pg1‚0.0009054351911026523 & lr-AdamW/pg2‚0.0009054351911026523 trainer/global_step‚2049 % _timestamp‚1.7885242838683064e+09 _runtime‚3522.089062937 _step‚163¯îÌGÌ´Æ % _timestamp‚1.7885242838683064e+09 _runtime‚3522.089062937 _step‚163 & lr-AdamW/pg1‚0.0009054351911026523 & lr-AdamW/pg2‚0.0009054351911026523 trainer/global_step‚2049£÷b–d2µ- _wandbruntime‚3522  _runtime‚3522£DÌ)c ¶ÁO E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚55406996 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚82605480882553 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4055641780736 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚78561096627216 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚38445307 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚86504140568064 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4055875448320 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4055641780736 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22888784 $ _timestamp‚1.788524283872535e+09 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14608987 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7mc¿Äd134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22888786 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚93852301 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚91509196166144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚55433338 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4126857263616 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚82448247116800 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚87382052476416 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚87382028536320 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4127143689728 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚86504140568064 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚82448247116800 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4055875448320 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4044384255337 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚669 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 _step‚164 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14609654 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚3522.093439039 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚91508885799936 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚86503888897536 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚82448265119744 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14609654 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚3522.093439039 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚91508885799936 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚86503888897536 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚82448265119744 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22888784 $ _timestamp‚1.788524283872535e+09 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14608987 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22888786 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚93852301 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚91509196166144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚55433338 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4126857263616 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚82448247116800 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚87382052476416 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360¤•®šw2¸- _wandbruntime‚3524  _runtime‚3524”×.–Ò&¹ÌM VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚93897268 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚93923638 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22899586 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚82487551156736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4057839487488 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37515280 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚86545390644224 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14616384 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚86545138960384 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 294801408 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚78598609945280 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4046113763247 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚55460056 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚86545390644224 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚37515972 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4057839487488 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22899588 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚78598626984640 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚93923638 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚38463580 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7885242861125777e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 _step‚165 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚38463580 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚82487533153792 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4057605806592 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4057605806592 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚55433688 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚82487551156736 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚93897268 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4129143625216 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4046342744531 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚09'³a©ºÑM VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚93897268 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚93923638 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22899586 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚82487551156736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4057839487488 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37515280 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚86545390644224 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14616384 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚86545138960384 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 294801408 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚78598609945280 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4046113763247 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚55460056 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚86545390644224 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚37515972 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4057839487488 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22899588 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚78598626984640 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚93923638 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚38463580 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7885242861125777e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 _step‚165 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚38463580 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚82487533153792 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4057605806592 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4057605806592 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚55433688 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚82487551156736 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚93897268 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4129143625216 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4046342744531 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0¥áÙ=2»- _wandbruntime‚3524  _runtime‚3524 õpX¼Ò /loss/FRt1/detection_loss‚2.6391552835702896 1loss/SRqqt3/detection_loss‚0.9846567587824319 /loss/FBt3/detection_loss‚3.5745540857315063 %loss/total_loss‚49.32390278749869 /loss/FBt4/detection_loss‚0.8462426206097007 % _timestamp‚1.7885242861207726e+09 0loss/FRt1/assignment_loss‚2.9508937038481236 /loss/FRt2/assignment_loss‚4.700717276997036 2loss/SRqqt3/assignment_loss‚1.2307320365564913 0loss/FBt1/assignment_loss‚0.4200236027502859 1loss/FBt4/assignment_loss‚0.39045451257475355 1loss/SRqqt1/detection_loss‚1.9158730924129486 0loss/FRt4/assignment_loss‚3.0268966775873434 .loss/FRt2/detection_loss‚3.469531112247043 0loss/SRqqt2/detection_loss‚2.615870631658114 1loss/SRqqt4/detection_loss‚2.7843817204236982 .loss/FBt1/detection_loss‚2.244189113378525 epoch‚1 trainer/global_step‚2049 _step‚166 2loss/SRqqt1/assignment_loss‚1.7265167558565735 1loss/SRqqt4/assignment_loss‚1.953619335591793 0loss/FBt2/assignment_loss‚0.2719060736136199 /loss/FRt4/detection_loss‚2.0422976395803625 .loss/FBt2/detection_loss‚2.543898300120705 _runtime‚3524.341321027 0loss/FRt3/assignment_loss‚1.7732604006305337 2loss/SRqqt2/assignment_loss‚2.2108144714282107 /loss/FBt3/assignment_loss‚1.086018684009711 .loss/FRt3/detection_loss‚1.921398897538893Ëf€€]½× /loss/FRt4/detection_loss‚2.0422976395803625 .loss/FBt2/detection_loss‚2.543898300120705 _runtime‚3524.341321027 0loss/FRt3/assignment_loss‚1.7732604006305337 2loss/SRqqt2/assignment_loss‚2.2108144714282107 /loss/FBt3/assignment_loss‚1.086018684009711 .loss/FRt3/detection_loss‚1.921398897538893 /loss/FRt1/detection_loss‚2.6391552835702896 1loss/SRqqt3/detection_loss‚0.9846567587824319 /loss/FBt3/detection_loss‚3.5745540857315063 %loss/total_loss‚49.32390278749869 /loss/FBt4/detection_loss‚0.8462426206097007 % _timestamp‚1.7885242861207726e+09 0loss/FRt1/assignment_loss‚2.9508937038481236 /loss/FRt2/assignment_loss‚4.700717276997036 2loss/SRqqt3/assignment_loss‚1.2307320365564913 0loss/FBt1/assignment_loss‚0.4200236027502859 1loss/FBt4/assignment_loss‚0.39045451257475355 1loss/SRqqt1/detection_loss‚1.9158730924129486 0loss/FRt4/assignment_loss‚3.0268966775873434 .loss/FRt2/detection_loss‚3.469531112247043 0loss/SRqqt2/detection_loss‚2.615870631658114 1loss/SRqqt4/detection_loss‚2.7843817204236982 .loss/FBt1/detection_loss‚2.244189113378525 epoch‚1 trainer/global_step‚2049 _step‚166 2loss/SRqqt1/assignment_loss‚1.7265167558565735 1loss/SRqqt4/assignment_loss‚1.953619335591793 0loss/FBt2/assignment_loss‚0.2719060736136199¦­f–Ñ›¾‚(: þíêÔ¹ÛÔø gpu.0.gpu‚18.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚99.556' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚19.911199999999997 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0mØ Œp¿‚(:å þíêÔ„­®ù network.sent‚ 4.561027e+06 disk.nvme0n1.in‚ 8185.2421875 disk.drbd1260.out‚0.484375 disk.dm-0.in‚ 8185.2421875 disk.sda.in‚0 disk.sda3.in‚0$ memory_percent‚99.95324611663818& proc.memory.availableMB‚ 7.66015625% disk./.usageGB‚1175.4604835510254' disk.nvme0c0n1.out‚67104.4873046875 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.sda2.out‚0" disk.dm-0.out‚67104.4873046875 disk.sda3.out‚ 101.34375" disk.nvme0c0n1.in‚ 8185.2421875% disk.nvme0n1.out‚67104.4873046875 disk.dm-1.out‚ 101.34375 disk.sda.out‚ 101.34375# proc.memory.rssMB‚ 2917.12890625* proc.memory.percent‚17.804741859436035 proc.cpu.threads‚17 network.recv‚ 1.125507e+06) disk./.usagePercent‚78.88613932814775 disk.dm-1.in‚0 disk.sda2.in‚0 cpu‚25.30408718835414A*"ÅœÀ‚(:‘ îêÔÿÏÖö gpu.0.gpu‚29.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚110.308' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.061600000000002 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ø¢džwÁ‚(:ì îêÔ¨«ì÷ disk.drbd1260.out‚ 0.48828125$ memory_percent‚99.95346069335938 network.recv‚ 1.13207e+06 disk.sda.out‚ 101.69140625 disk.sda3.in‚0 disk.sda3.out‚ 101.69140625" disk.nvme0c0n1.in‚ 8185.2421875 disk.nvme0n1.in‚ 8185.2421875& disk.nvme0n1.out‚67409.84716796875 disk.sda1.in‚0 network.sent‚ 4.66931e+06 disk.dm-0.in‚ 8185.2421875# disk.dm-0.out‚67409.84716796875( disk.nvme0c0n1.out‚67409.84716796875 disk.drbd1260.in‚0# proc.memory.rssMB‚ 2917.12890625* proc.memory.percent‚17.804741859436035 cpu‚25.303813948353564 disk.sda2.out‚0 disk.sda1.out‚0! proc.memory.availableMB‚7.625 proc.cpu.threads‚17( disk./.usagePercent‚78.8877142864984% disk./.usageGB‚1175.4839515686035 disk.sda2.in‚0 disk.sda.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 101.69140625.4Xd‘‚(:† œîêÔÁãÔö gpu.0.gpu‚29.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚112.848' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.5696 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 wÂ(:ì œîêÔ¬¶’ø# proc.memory.availableMB‚7.84375# proc.memory.rssMB‚ 2917.12890625* proc.memory.percent‚17.804741859436035 cpu‚25.303724360646147 disk.nvme0n1.in‚ 8185.2421875& disk.nvme0n1.out‚67750.95263671875 network.sent‚ 4.679206e+06 disk.sda3.in‚0( disk.nvme0c0n1.out‚67750.95263671875 disk.drbd1260.in‚0 disk.sda2.in‚0# memory_percent‚99.9521255493164 disk.dm-0.in‚ 8185.2421875 disk.dm-1.in‚0 disk.dm-1.out‚ 102.0703125 proc.cpu.threads‚17 network.recv‚ 1.144084e+06# disk.dm-0.out‚67750.95263671875 disk.sda.in‚0 disk.sda.out‚ 102.0703125 disk.sda3.out‚ 102.0703125 disk.sda1.in‚0 disk.drbd1260.out‚ 0.48828125 disk.sda2.out‚0( disk./.usagePercent‚78.8877142864984% disk./.usageGB‚1175.4839515686035" disk.nvme0c0n1.in‚ 8185.2421875 disk.sda1.out‚0gZÖ±œÄ‚(:‘ «îêÔÙ‰·ö gpu.0.gpu‚33.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚132.932' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.586399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ü|ÑlÅ‚(:á «îêÔÄ‹·ù% disk./.usageGB‚1175.4839515686035 disk.sda2.in‚0 disk.sda2.out‚0" disk.dm-0.out‚67905.4267578125 disk.sda.in‚0 disk.sda.out‚ 102.578125 disk.dm-0.in‚ 8339.734375 disk.nvme0n1.in‚ 8339.734375 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0* proc.memory.percent‚17.804741859436035 cpu‚25.30341448590972 proc.cpu.threads‚17! disk.nvme0c0n1.in‚ 8339.734375 network.sent‚ 4.684242e+06 network.recv‚ 1.147294e+06 disk.drbd1260.out‚ 0.4921875# memory_percent‚99.9521255493164( disk./.usagePercent‚78.8877142864984' disk.nvme0c0n1.out‚67905.4267578125% disk.nvme0n1.out‚67905.4267578125 disk.dm-1.out‚ 102.578125 disk.drbd1260.in‚0# proc.memory.availableMB‚7.84375# proc.memory.rssMB‚ 2917.12890625 disk.sda3.in‚0 disk.sda3.out‚ 102.578125TÜ(’‘Æ‚(:† ºîêÔÄŠ•÷ gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚116.881' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.3762 gpu.0.smClock‚1290 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0&œ}(nÇ‚(:ã ºîêÔù" disk.dm-0.out‚68054.2822265625 disk.sda3.out‚ 102.859375$ memory_percent‚99.95365142822266 proc.cpu.threads‚17 network.recv‚ 1.151092e+06( disk./.usagePercent‚78.8877142864984% disk./.usageGB‚1175.4839515686035' disk.nvme0c0n1.out‚68054.2822265625 disk.sda1.out‚0 disk.dm-1.out‚ 102.859375 disk.drbd1260.out‚ 0.4921875 disk.sda2.in‚0 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda2.out‚0 disk.sda.in‚0 disk.sda.out‚ 102.859375 disk.sda3.in‚0# proc.memory.availableMB‚7.59375# proc.memory.rssMB‚ 2917.12890625 network.sent‚ 4.689373e+06 disk.dm-0.in‚ 8339.734375* proc.memory.percent‚17.804741859436035 cpu‚25.303126626520065! disk.nvme0c0n1.in‚ 8339.734375 disk.nvme0n1.in‚ 8339.734375% disk.nvme0n1.out‚68054.2822265625 disk.sda1.in‚0*Ÿ¿2È- _wandbruntime‚3599  _runtime‚3599xŠ(ðÅÉ¿ trainer/global_step‚2099 % _timestamp‚1.7885243617627792e+09 _runtime‚3599.982429268 _step‚167 % lr-AdamW/pg1‚0.000903795924158514 % lr-AdamW/pg2‚0.000903795924158514D‚#_ÊÊÄ % _timestamp‚1.7885243617627792e+09 _runtime‚3599.982429268 _step‚167 % lr-AdamW/pg1‚0.000903795924158514 % lr-AdamW/pg2‚0.000903795924158514 trainer/global_step‚2099§^è&52Ë- _wandbruntime‚3599  _runtime‚3599ßj©È'ÌÂO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚56744948 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23428886 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚93693180737024 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚88566658124288 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚704 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4226929758208 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚80437614498416 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚706 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4226628652032 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚84412548966400 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 _runtime‚3599.983573304 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14958060 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚88566406453760 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚84579936794261 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚39359055 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚84412548966400 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚56771290 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚56744948 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚84412566969344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14957356 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4154091154944 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚89466227038720 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚88566406453760 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4153857487360 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 trainer/global_step‚2099 _step‚168 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4153857487360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚56771290 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39359057 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 325046272 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚89466250978816 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 301106176 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚84579690773825 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4154091154944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38386946 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39359057 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 325046272 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚89466250978816 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 301106176 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚84579690773825 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4154091154944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38386946 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚84412548966400 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 _runtime‚3599.983573304 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14958060 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚88566406453760 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚84579936794261 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚39359055 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚84412548966400 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚56771290 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚56744948 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚84412566969344 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14957356 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582¨°"\uœÎ‚(:‘ ÉîêÔ¨›•ø gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚114.594' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.918799999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0kg:¤pÏ‚(:å ÉîêÔ“·£ø cpu‚25.302718098157357$ disk./.usageGB‚1175.460521697998 disk.nvme0n1.in‚ 8339.734375$ disk.nvme0n1.out‚68329.212890625 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.sda3.out‚ 103.24609375# memory_percent‚99.9521255493164 network.recv‚ 1.155326e+06 disk.sda1.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 8339.734375* proc.memory.percent‚17.804741859436035 proc.cpu.threads‚17 disk.drbd1260.out‚ 0.49609375( disk./.usagePercent‚78.8861418882231! disk.nvme0c0n1.in‚ 8339.734375& disk.nvme0c0n1.out‚68329.212890625 disk.sda.in‚0 disk.sda.out‚ 103.24609375# proc.memory.rssMB‚ 2917.12890625 network.sent‚ 4.695219e+06 disk.sda1.out‚0 disk.dm-1.out‚ 103.24609375! disk.dm-0.out‚68329.212890625 disk.sda3.in‚0# proc.memory.availableMB‚7.84375¡YS—2Ð- _wandbruntime‚3600  _runtime‚3600_¤M¿Ò&ÑÌM VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚668 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚84451853006336 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚84451853006336 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 _step‚169 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚88607656516608 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 % _timestamp‚1.7885243626800067e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚96175340 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚56771640 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39377332 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38404473 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4228633307136 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4144280785039 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚38405141 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306158592 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚80475127816480 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4155821513216 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚96148970 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚96175340 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23439688 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚88607908200448 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚88607908200448 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚56798008 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14964787 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4156055194112 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚96148970 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39377332 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚56798008 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚666 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚3600.902515602 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚84619425640879 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚80475144855840 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚56771640 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14965453 trainer/global_step‚2099 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4156055194112 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4155821513216 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚89507911008768 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚93736850474496 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚93736544315904 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896úDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚96175340 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23439688 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚88607908200448 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚88607908200448 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚56798008 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14964787 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4156055194112 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚96148970 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39377332 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚56798008 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚666 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚3600.902515602 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚84619425640879 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚80475144855840 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚56771640 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14965453 trainer/global_step‚2099 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4156055194112 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4155821513216 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚89507911008768 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚93736850474496 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚93736544315904 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚668 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚84451853006336 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚84451853006336 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 _step‚169 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚88607656516608 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 % _timestamp‚1.7885243626800067e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚96175340 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚56771640 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39377332 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38404473 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4228633307136 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4144280785039 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚38405141 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273©Õ.ÊÝ2Ó- _wandbruntime‚3600  _runtime‚3600Ë°þÜVÔÐ 0loss/FRt3/assignment_loss‚2.1952237591550157 1loss/SRqqt4/detection_loss‚2.8288382134939494 .loss/FBt3/detection_loss‚2.628819987177849 /loss/FBt4/detection_loss‚0.6693559434750805 % _timestamp‚1.7885243626832094e+09 _runtime‚3600.905648135 0loss/SRqqt2/assignment_loss‚1.91253671169281 1loss/FBt2/assignment_loss‚0.19820416271139152 .loss/FRt1/detection_loss‚2.817472340968939 .loss/FRt4/detection_loss‚1.943570208158749 /loss/FBt2/detection_loss‚2.4141854494810104 -loss/FRt1/assignment_loss‚4.1356460750103 /loss/FRt2/assignment_loss‚4.778998576677763 0loss/FRt4/assignment_loss‚2.9974044548428576 0loss/SRqqt4/assignment_loss‚2.05916460093699 1loss/FBt4/assignment_loss‚0.19182550650524294 1loss/SRqqt2/detection_loss‚2.5299591779708863 1loss/SRqqt3/detection_loss‚1.2037484099467595 %loss/total_loss‚49.20161551405036 1loss/FBt1/assignment_loss‚0.45050712617762656 1loss/FBt3/assignment_loss‚0.28090924862772226 .loss/FRt2/detection_loss‚4.216060798901778 epoch‚1 trainer/global_step‚2099 .loss/FBt1/detection_loss‚2.423211097717285 /loss/FRt3/detection_loss‚1.7048264304427683 2loss/SRqqt1/assignment_loss‚1.5657510812515798 2loss/SRqqt3/assignment_loss‚1.0770163680071987 1loss/SRqqt1/detection_loss‚1.9783797847188038 _step‚170¸Ã[ÕÕ /loss/FRt3/detection_loss‚1.7048264304427683 2loss/SRqqt1/assignment_loss‚1.5657510812515798 2loss/SRqqt3/assignment_loss‚1.0770163680071987 1loss/SRqqt1/detection_loss‚1.9783797847188038 _step‚170 0loss/FRt3/assignment_loss‚2.1952237591550157 1loss/SRqqt4/detection_loss‚2.8288382134939494 .loss/FBt3/detection_loss‚2.628819987177849 /loss/FBt4/detection_loss‚0.6693559434750805 % _timestamp‚1.7885243626832094e+09 _runtime‚3600.905648135 0loss/SRqqt2/assignment_loss‚1.91253671169281 1loss/FBt2/assignment_loss‚0.19820416271139152 .loss/FRt1/detection_loss‚2.817472340968939 .loss/FRt4/detection_loss‚1.943570208158749 /loss/FBt2/detection_loss‚2.4141854494810104 -loss/FRt1/assignment_loss‚4.1356460750103 /loss/FRt2/assignment_loss‚4.778998576677763 0loss/FRt4/assignment_loss‚2.9974044548428576 0loss/SRqqt4/assignment_loss‚2.05916460093699 1loss/FBt4/assignment_loss‚0.19182550650524294 1loss/SRqqt2/detection_loss‚2.5299591779708863 1loss/SRqqt3/detection_loss‚1.2037484099467595 %loss/total_loss‚49.20161551405036 1loss/FBt1/assignment_loss‚0.45050712617762656 1loss/FBt3/assignment_loss‚0.28090924862772226 .loss/FRt2/detection_loss‚4.216060798901778 epoch‚1 trainer/global_step‚2099 .loss/FBt1/detection_loss‚2.423211097717285ª½±‘Ö‚(:† ØîêÔõäèõ gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚138.873' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.7746 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0·Sutׂ(:é ØîêÔ³¨›ù$ memory_percent‚99.95086193084717& proc.memory.availableMB‚ 8.05078125# proc.memory.rssMB‚ 2917.12890625 network.recv‚ 1.163188e+06$ disk.nvme0n1.out‚68646.349609375 disk.sda1.in‚0 disk.dm-1.in‚0 disk.drbd1260.in‚0* proc.memory.percent‚17.804741859436035 cpu‚25.302557694804896 proc.cpu.threads‚17 network.sent‚ 4.793476e+06( disk./.usagePercent‚78.8861418882231& disk.nvme0c0n1.out‚68646.349609375 disk.nvme0n1.in‚ 8339.734375$ disk./.usageGB‚1175.460521697998! disk.nvme0c0n1.in‚ 8339.734375 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda.in‚0 disk.sda.out‚ 103.91796875 disk.sda3.in‚0 disk.dm-1.out‚ 103.91796875 disk.drbd1260.out‚ 0.49609375 disk.dm-0.in‚ 8339.734375! disk.dm-0.out‚68646.349609375 disk.sda3.out‚ 103.91796875yk*$œØ‚(:‘ çîêÔÂœõö gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚113.533' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.706599999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0’F‚pÙ‚(:å çîêÔ궇ø network.sent‚ 4.893732e+06 network.recv‚ 1.175201e+06' disk.nvme0c0n1.out‚68872.8486328125 disk.drbd1260.out‚0.5 disk.sda2.in‚0 disk.sda3.out‚ 104.36328125( disk./.usagePercent‚78.8877209426943! disk.nvme0c0n1.in‚ 8487.546875 disk.nvme0n1.in‚ 8487.546875 disk.dm-1.out‚ 104.36328125 disk.drbd1260.in‚0 disk.sda2.out‚0 disk.sda.in‚0 disk.sda.out‚ 104.36328125% disk.nvme0n1.out‚68872.8486328125 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 8487.546875% proc.memory.availableMB‚ 8.2890625# proc.memory.rssMB‚ 2917.12890625 proc.cpu.threads‚17% disk./.usageGB‚1175.4840507507324 disk.dm-1.in‚0" disk.dm-0.out‚68872.8486328125 disk.sda3.in‚0$ memory_percent‚99.94940757751465* proc.memory.percent‚17.804741859436035 cpu‚25.302427303871667&,œÚ‚(:‘ öîêÔ¡çÔö gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚133.087' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.617399999999996 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ؼäSoÛ‚(:ä öîêÔ—«ûø# proc.memory.rssMB‚ 2917.12890625 cpu‚25.302133983094855 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.sda.out‚ 104.81640625! disk.nvme0c0n1.in‚ 8487.546875 disk.nvme0n1.in‚ 8487.546875* proc.memory.percent‚17.804741859436035 proc.cpu.threads‚17 disk.drbd1260.out‚0.5 disk.sda2.out‚0 disk.dm-0.in‚ 8487.546875" disk.dm-0.out‚69099.3681640625 disk.sda.in‚0 disk.sda1.in‚0% proc.memory.availableMB‚ 8.0390625 network.recv‚ 1.178713e+06( disk./.usagePercent‚78.8877209426943 disk.sda3.out‚ 104.81640625 disk.dm-1.in‚0 disk.dm-1.out‚ 104.81640625# memory_percent‚99.9509334564209 network.sent‚ 4.898836e+06% disk./.usageGB‚1175.4840507507324 disk.sda3.in‚0' disk.nvme0c0n1.out‚69099.3681640625% disk.nvme0n1.out‚69099.3681640625 disk.sda1.out‚0"K‘Ü‚(:† …ïêÔ«³Œö gpu.0.gpu‚11.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚117.838' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.5676 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0yjZånÝ‚(:ã …ïêԤіø network.recv‚ 1.183447e+06$ disk./.usageGB‚1175.484001159668$ disk.nvme0c0n1.out‚ 69440.6171875 disk.dm-1.out‚ 105.11328125 disk.sda2.out‚0 disk.dm-0.in‚ 8487.546875 disk.sda.in‚0* proc.memory.percent‚17.804741859436035 disk.sda1.in‚0 disk.dm-1.in‚0 disk.sda2.in‚0 disk.sda.out‚ 105.11328125 disk.sda3.in‚0 disk.sda3.out‚ 105.11328125# proc.memory.rssMB‚ 2917.12890625 cpu‚25.302001907430935) disk./.usagePercent‚78.88771761459634 disk.nvme0n1.in‚ 8487.546875" disk.nvme0n1.out‚ 69440.6171875 disk.sda1.out‚0 disk.drbd1260.out‚ 0.50390625& proc.memory.availableMB‚ 8.04296875 proc.cpu.threads‚17 network.sent‚ 4.90464e+06! disk.nvme0c0n1.in‚ 8487.546875 disk.drbd1260.in‚0 disk.dm-0.out‚ 69440.6171875$ memory_percent‚99.95090961456299é,‘Þ‚(:† ”ïêÔ®ì­ö gpu.0.gpu‚30.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚136.146' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.2292 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚08À¯ß‚(:á ”ïêÔÆŒ©ù# disk.nvme0c0n1.out‚ 69498.859375 disk.dm-1.in‚0 disk.dm-1.out‚ 105.48828125$ memory_percent‚99.95090961456299 cpu‚25.30170948833709! disk.nvme0c0n1.in‚ 8487.546875 disk.nvme0n1.in‚ 8487.546875! disk.nvme0n1.out‚ 69498.859375 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 2917.12890625* proc.memory.percent‚17.804741859436035) disk./.usagePercent‚78.88771915064156 disk.dm-0.out‚ 69498.859375 disk.sda3.in‚0& proc.memory.availableMB‚ 8.04296875 proc.cpu.threads‚17 network.sent‚ 4.918862e+06 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.5`ñ쎽0390625 disk.dm-0.in‚ 8487.546875 disk.sda.in‚0 disk.sda.out‚ 105.48828125 disk.sda3.out‚ 105.48828125 network.recv‚ 1.203175e+06% disk./.usageGB‚1175.4840240478516¨~$B2à- _wandbruntime‚3679  _runtime‚3679«‡*ƒÇáÁ & lr-AdamW/pg1‚0.0009020640542206461 & lr-AdamW/pg2‚0.0009020640542206461 trainer/global_step‚2149 % _timestamp‚1.7885244410711553e+09 _runtime‚3679.293053358 _step‚171y=ÓøÌâÆ _step‚171 & lr-AdamW/pg1‚0.0009020640542206461 & lr-AdamW/pg2‚0.0009020640542206461 trainer/global_step‚2149 % _timestamp‚1.7885244410711553e+09 _runtime‚3679.293053358«Ü ½2ã- _wandbruntime‚3679  _runtime‚3679£ ‘xÇ'äÁO H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 $ _timestamp‚1.788524441073321e+09 _runtime‚3679.294381039 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚90629175680512 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚86376868818944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 290620416 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚58082900 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚90628924009984 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚58109242 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚674 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚95876784178688 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15306017 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚39275001 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚91550449481216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚86554146685533 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚39275675 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚82314132369616 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚90628924009984 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚95877098739200 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚40272807 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4252306861568 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚91550425541120 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4326358637568 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4240031355277 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4252073193984 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 trainer/global_step‚2149 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4326649257984 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4252073193984 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚98382049 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4240260336353 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚98355705 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚86376850816000 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15306689 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136‹4âÌ'åÆO OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 $ _timestamp‚1.788524441073321e+09 _runtime‚3679.294381039 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚90629175680512 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚86376868818944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 290620416 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚58082900 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚90628924009984 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚58109242 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚674 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚95876784178688 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15306017 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚39275001 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚91550449481216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚86554146685533 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚39275675 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚82314132369616 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚90628924009984 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚95877098739200 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚40272807 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4252306861568 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚91550425541120 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4326358637568 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4240031355277 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4252073193984 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 trainer/global_step‚2149 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4326649257984 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4252073193984 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚98382049 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4240260336353 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚98355705 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚86376850816000 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15306689 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0¬ñVû2æ- _wandbruntime‚3680  _runtime‚3680€ˆ"Ñ&çËM \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4254270900736 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚86593881552587 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4328641800192 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4242218825547 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4241989844263 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚58109592 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚86593635531943 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40291080 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4254270900736 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚721 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4254037219840 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚39293868 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40291082 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39293145 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 trainer/global_step‚2149 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4254037219840 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4328351193088 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 _runtime‚ 3680.30275384 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40291080 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚98427042 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚86416154855936 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23979786 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚95920460704256 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚58135960 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚86416154855936 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚90670425756672 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 % _timestamp‚1.7885244420820472e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15313359 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚86416136852992 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15314080 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚98400672 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314547200 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚173 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚723 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚82351662727040 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚86416136852992 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚95920775251456 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚98427042 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309¢rìpèÐM ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚91592109511168 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚173 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚723 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚82351662727040 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚86416136852992 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚95920775251456 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚98427042 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4254270900736 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚86593881552587 OADeviceStatsMonitor.on_train_batw|¶f ch_end/requested_bytes.all.current‚ 246020644 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4328641800192 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4242218825547 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4241989844263 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚58109592 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚86593635531943 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40291080 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4254270900736 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚721 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4254037219840 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚39293868 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40291082 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39293145 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 trainer/global_step‚2149 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4254037219840 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4328351193088 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 _runtime‚ 3680.30275384 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40291080 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚98427042 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚86416154855936 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23979786 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚95920460704256 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚58135960 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚86416154855936 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚90670425756672 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 % _timestamp‚1.7885244420820472e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15313359 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚86416136852992 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15314080 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚98400672 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314547200 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OÑ‹]ê× 0loss/FBt3/assignment_loss‚0.4181748871225864 /loss/FRt1/assignment_loss‚3.684995256937467 1loss/SRqqt4/detection_loss‚2.5151497406118057 /loss/FBt1/detection_loss‚2.2286075320508747 /loss/FBt2/detection_loss‚3.0918963154157004 /loss/FBt4/detection_loss‚0.9470025629959475 epoch‚1 2loss/SRqqt2/assignment_loss‚2.1288324362039566 0loss/FBt1/assignment_loss‚0.4455837909432982 1loss/FBt2/assignment_loss‚0.16661496974147666 /loss/FBt3/detection_loss‚2.7688652992248537 _step‚174 2loss/SRqqt1/assignment_loss‚1.6728090468095616 .loss/FRt3/detection_loss‚1.659951000213623 /loss/FRt4/detection_loss‚1.7195562219120941 1loss/SRqqt1/detection_loss‚2.0728847393766046 /loss/FRt1/detection_loss‚3.1771787932285895 &loss/total_loss‚49.887152474794846 2loss/SRqqt3/assignment_loss‚1.1902810965969772 trainer/global_step‚2149 % _timestamp‚1.7885244420843308e+09 _runtime‚3680.303744947 /loss/FRt2/assignment_loss‚4.868306960378375 /loss/FRt3/assignment_loss‚2.464704215526581 0loss/FRt4/assignment_loss‚3.6461682380256004 2loss/SRqqt4/assignment_loss‚2.2242599848438713 1loss/FBt4/assignment_loss‚0.16691470920234638 .loss/FRt2/detection_loss‚3.202185727301098 1loss/SRqqt2/detection_loss‚2.3315273356437682 1loss/SRqqt3/detection_loss‚1.09470161448778777<>ÑbëÜ 2loss/SRqqt2/assignment_loss‚2.1288324362039566 0loss/FBt1/assignment_loss‚0.4455837909432982 1loss/FBt2/assignment_loss‚0.16661496974147666 /loss/FBt3/detection_loss‚2.7688652992248537 _step‚174 2loss/SRqqt1/assignment_loss‚1.6728090468095616 .loss/FRt3/detection_loss‚1.659951000213623 /loss/FRt4/detection_loss‚1.7195562219120941 1loss/SRqqt1/detection_loss‚2.0728847393766046 /loss/FRt1/detection_loss‚3.1771787932285895 &loss/total_loss‚49.887152474794846 2loss/SRqqt3/assignment_loss‚1.1902810965969772 trainer/global_step‚2149 % _timestamp‚1.7885244420843308e+09 _runtime‚3680.303744947 /loss/FRt2/assignment_loss‚4.868306960378375 /loss/FRt3/assignment_loss‚2.464704215526581 0loss/FRt4/assignment_loss‚3.6461682380256004 2loss/SRqqt4/assignment_loss‚2.2242599848438713 1loss/FBt4/assignment_loss‚0.16691470920234638 .loss/FRt2/detection_loss‚3.202185727301098 1loss/SRqqt2/detection_loss‚2.3315273356437682 1loss/SRqqt3/detection_loss‚1.0947016144877877 0loss/FBt3/assignment_loss‚0.4181748871225864 /loss/FRt1/assignment_loss‚3.684995256937467 1loss/SRqqt4/detection_loss‚2.5151497406118057 /loss/FBt1/detection_loss‚2.2286075320508747 /loss/FBt2/detection_loss‚3.0918963154157004 /loss/FBt4/detection_loss‚0.9470025629959475 epoch‚1®¬§Sì‚(:„ £ïêÔ´Ò¼ú gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚92.852' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.5704 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 *5yí‚(:î £ïêÔ–ãý network.recv‚ 1.206532e+06( disk./.usagePercent‚78.8877158225436$ disk./.usageGB‚1175.483974456787 disk.sda.in‚0# disk.nvme0c0n1.in‚ 8611.31640625& disk.nvme0c0n1.out‚69816.306640625 disk.drbd1260.out‚ 0.5078125 disk.sda2.out‚0 disk.dm-0.in‚ 8611.31640625 disk.sda3.in‚0 disk.sda3.out‚ 105.90234375 disk.dm-1.out‚ 105.90234375 disk.drbd1260.in‚0$ memory_percent‚99.95286464691162& proc.memory.availableMB‚ 7.72265625* proc.memory.percent‚17.804741859436035 network.sent‚ 4.923681e+06! disk.dm-0.out‚69816.306640625 disk.sda.out‚ 105.90234375! disk.nvme0n1.in‚ 8611.31640625$ disk.nvme0n1.out‚69816.306640625 disk.sda2.in‚0# proc.memory.rssMB‚ 2917.12890625 cpu‚25.301449869568852 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0 proc.cpu.threads‚17I9prî‚(:ç ²ïêÔŽÓŒø disk.sda.in‚0 disk.sda.out‚ 106.3515625 disk.sda3.in‚0 disk.sda3.out‚ 106.3515625 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.5078125# disk.nvme0c0n1.in‚ 8611.31640625! disk.nvme0n1.in‚ 8611.31640625 disk.sda1.in‚0$ memory_percent‚99.95193481445312! proc.memory.availableMB‚7.875# proc.memory.rssMB‚ 2917.12890625* proc.memory.percent‚17.804741859436035 network.sent‚ 5.030984e+06 network.recv‚ 1.212088e+06) disk./.usagePercent‚78.79744807798555% disk./.usageGB‚1174.1389198303223& disk.nvme0c0n1.out‚69946.685546875$ disk.nvme0n1.out‚69946.685546875 disk.dm-1.in‚0 cpu‚25.30071439796182 disk.sda2.out‚0 disk.dm-0.in‚ 8611.31640625! disk.dm-0.out‚69946.685546875 disk.dm-1.out‚ 106.3515625 proc.cpu.threads‚17 disk.sda2.in‚0ŸTœï‚(:‘ ²ïêÔªÒù gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚33.0 gpu.0.powerWatts‚145.569' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.113799999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0œÔÂpð‚(:å ÁïêÔ„Ÿø÷ network.sent‚ 5.046801e+06) disk./.usagePercent‚78.80072574244643 disk.sda2.out‚0! disk.nvme0n1.in‚ 8611.31640625 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.51171875# proc.memory.rssMB‚ 2917.12890625* proc.memory.percent‚17.804741859436035 disk.dm-0.in‚ 8611.31640625 disk.sda.in‚0 disk.sda.out‚ 106.6796875 disk.sda3.out‚ 106.6796875 disk.dm-1.out‚ 106.6796875 disk.drbd1260.in‚0% proc.memory.availableMB‚ 6.2890625 cpu‚25.29991826105378 network.recv‚ 1.236317e+06$ disk./.usageGB‚1174.187759399414 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 8611.31640625 disk.sda1.in‚0 proc.cpu.threads‚17 disk.dm-0.out‚ 70330.2890625 disk.sda3.in‚0$ disk.nvme0c0n1.out‚ 70330.2890625" disk.nvme0n1.out‚ 70330.2890625 disk.sda1.out‚0$ memory_percent‚99.96161460876465 Y -‘ñ‚(:† ÁïêÔ‘ú ú gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚33.0 gpu.0.powerWatts‚122.504' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.5008 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0]ÃØ¿pò‚(:å ÐïêÔ™Ûø disk.dm-0.in‚ 8611.31640625# disk.nvme0c0n1.in‚ 8611.31640625 disk.sda1.in‚0$ memory_percent‚99.95689392089844 disk.sda.out‚ 107.0390625% disk.nvme0c0n1.out‚70498.76953125# disk.nvme0n1.out‚70498.76953125 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.51171875 proc.cpu.threads‚17 network.sent‚ 5.059705e+06 network.recv‚ 1.25413e+06) disk./.usagePercent‚78.80059594662656 disk.sda.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 107.0390625 disk.dm-1.in‚0 disk.dm-1.out‚ 107.0390625 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.out‚70498.76953125! disk.nvme0n1.in‚ 8611.31640625" proc.memory.availableMB‚7.0625# proc.memory.rssMB‚ 2917.12890625* proc.memory.percent‚17.804741859436035 cpu‚25.29905743473459% disk./.usageGB‚1174.1858253479004‘ó‚(:† ÐïêÔšÙ§ù gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚115.673' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.1346 gpu.0.smClock‚1305 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0r½W2ô- _wandbruntime‚3737  _runtime‚3737¯{ ÇõÁ & lr-AdamW/pg1‚0.0009002399422081479 & lr-AdamW/pg2‚0.0009002399422081479 trainer/global_step‚2199 % _timestamp‚1.7885244989140809e+09 _runtime‚3737.133749048 _step‚175Wy…ÌöÆ & lr-AdamW/pg1‚0.0009002399422081479 & lr-AdamW/pg2‚0.0009002399422081479 trainer/global_step‚2199 % _timestamp‚1.7885244989140809e+09 _runtime‚3737.133749048 _step‚175¯ÊÎ^2÷- _wandbruntime‚3737  _runtime‚37372@«Ì'øÆO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 100607407 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4350288900608 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚41186555 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚98061025045504 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚88341170668544 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚40167306 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚92691693236736 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚88528602597241 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 _runtime‚3737.135113508 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚88341152665600 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4337969395785 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚41186555 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 % _timestamp‚1.7885244989153087e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚59447194 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚682 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4350522568192 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚93634624043520 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4426084344320 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4338198376861 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚92691441566208 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24509086 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292717568 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496„–B¸PùËO ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4337969395785 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚41186555 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 % _timestamp‚1.7885244989153087e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚59447194 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚682 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4350522568192 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚93634624043520 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4426084344320 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4338198376861 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚92691441566208 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24509086 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292717568 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 100607407 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4350288900608 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚41186555 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚98061025045504 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚88341170668544 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚40167306 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚92691693236736 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚88528602597241 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 _runtime‚3737.135113508 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚88341152665600 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532°4S¢Ì2ú- _wandbruntime‚3738  _runtime‚3738,»’Ö&ûÐM % _timestamp‚1.7885245000199683e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚98104389996544 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 _step‚177 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚88380438702592 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚59473912 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚40184695 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚93676331953664 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚88568091443651 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚59473912 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚59447544 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚88380438702592 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4352486607360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚665 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚98104700349440 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚41204832 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 100678744 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚88380456705536 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚2199 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚92732943312896 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4352252926464 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 100652374 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚667 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286412800 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4352252926464 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310352896 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 100652374 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4428081982976 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4339927884771 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 100678744 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40184028 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24519886 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚88380456705536 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608í»4lÛ&üÕM VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚92732943312896 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4352252926464 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 100652374 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚667 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286412800 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4352252926464 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310352896 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 100652374 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4428081982976 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4339927884771 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 100678744 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40184028 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24519886 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚88380456705536 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 % _timestamp‚1.7885245000199683e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚98104389996544 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 _step‚177 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚88380438702592 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚59473912 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚40184695 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚93676331953664 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚88568091443651 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚59473912 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚59447544 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚88380438702592 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4352486607360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚665 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚98104700349440 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚41204832 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 100678744 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚88380456705536 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚2199±Ï’ÈÖ2ý- _wandbruntime‚3738  _runtime‚3738*2Ì4UþÏ 1loss/SRqqt1/assignment_loss‚1.726623362831531 1loss/SRqqt3/assignment_loss‚1.207069960112373 /loss/FBt1/assignment_loss‚0.492092568539083 /loss/FRt3/detection_loss‚1.6427620851434768 /loss/FBt1/detection_loss‚2.2334871387481687 .loss/FBt3/detection_loss‚1.687523705618722 1loss/SRqqt2/detection_loss‚2.6908388495445252 1loss/SRqqt3/detection_loss‚1.1466931574569459 &loss/total_loss‚49.707940837753156 0loss/FBt3/assignment_loss‚0.5133845937172217 .loss/FRt1/detection_loss‚2.942638317743937 /loss/FRt2/assignment_loss‚5.083020220200221 0loss/FRt4/assignment_loss‚3.6387435478984185 1loss/FBt2/assignment_loss‚0.15670902072983495 0loss/SRqqt1/detection_loss‚2.329877991830149 _runtime‚3738.242317205 /loss/FRt1/assignment_loss‚4.097343077262242 /loss/FBt2/detection_loss‚2.8669864028692245 /loss/FBt4/detection_loss‚1.0262949530608378 epoch‚1 % _timestamp‚1.7885245000227072e+09 1loss/SRqqt4/assignment_loss‚2.283528829162771 1loss/FBt4/assignment_loss‚0.29638759832702966 /loss/FRt4/detection_loss‚1.9731690860792714 0loss/SRqqt4/detection_loss‚2.414411177689379 trainer/global_step‚2199 -loss/FRt2/detection_loss‚3.26190019150575 2loss/SRqqt2/assignment_loss‚1.6707302665710448 _step‚178 /loss/FRt3/assignment_loss‚2.325724735110998š-o³ZÿÔ 0loss/FBt3/assignment_loss‚0.5133845937172217 .loss/FRt1/detection_loss‚2.942638317743937 /loss/FRt2/assignment_loss‚5.083020220200221 0loss/FRt4/assignment_loss‚3.6387435478984185 1loss/FBt2/assignment_loss‚0.15670902072983495 0loss/SRqqt1/detection_loss‚2.329877991830149 _runtime‚3738.242317205 /loss/FRt1/assignment_loss‚4.097343077262242 /loss/FBt2/detection_loss‚2.8669864028692245 /loss/FBt4/detection_loss‚1.0262949530608378 epoch‚1 % _timestamp‚1.7885245000227072e+09 1loss/SRqqt4/assignment_loss‚2.283528829162771 1loss/FBt4/assignment_loss‚0.29638759832702966 /loss/FRt4/detection_loss‚1.9731690860792714 0loss/SRqqt4/detection_loss‚2.414411177689379 trainer/global_step‚2199 -loss/FRt2/detection_loss‚3.26190019150575 2loss/SRqqt2/assignment_loss‚1.6707302665710448 _step‚178 /loss/FRt3/assignment_loss‚2.325724735110998 1loss/SRqqt1/assignment_loss‚1.726623362831531 1loss/SRqqt3/assignment_loss‚1.207069960112373 /loss/FBt1/assignment_loss‚0.492092568539083 /loss/FRt3/detection_loss‚1.6427620851434768 /loss/FBt1/detection_loss‚2.2334871387481687 .loss/FBt3/detection_loss‚1.687523705618722 1loss/SRqqt2/detection_loss‚2.6908388495445252 1loss/SRqqt3/detection_loss‚1.1466931574569459 &loss/total_loss‚49.707940837753156²¨ªŸœ€‚(:‘ ßïêÔú¦‹÷ gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚112.323' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.464599999999997 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Žë ®t‚(:é ßïêÔºõ‹ù disk.nvme0c0n1.in‚ 8641.53125 disk.sda1.in‚0 disk.drbd1260.out‚0.515625 disk.dm-0.in‚ 8641.53125# disk.dm-0.out‚70590.99951171875# proc.memory.rssMB‚ 2917.12890625 disk.sda.out‚ 107.43359375( disk.nvme0c0n1.out‚70590.99951171875 disk.dm-1.out‚ 107.43359375 disk.sda2.out‚0* proc.memory.percent‚17.804741859436035 network.recv‚ 1.257714e+06) disk./.usagePercent‚78.80072574244643 disk.sda3.in‚0 disk.sda3.out‚ 107.43359375 disk.drbd1260.in‚0 cpu‚25.298192712359114 proc.cpu.threads‚17 network.sent‚ 5.064828e+06 disk.nvme0n1.in‚ 8641.53125& disk.nvme0n1.out‚70590.99951171875 disk.sda1.out‚0 disk.dm-1.in‚0 disk.sda2.in‚0$ memory_percent‚99.95565414428711$ proc.memory.availableMB‚7.265625$ disk./.usageGB‚1174.187759399414 disk.sda.in‚0–»á÷‚‚(:… îïêÔˆÖ¸÷ gpu.0.gpu‚5.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚101.314' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.2628 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ѨKtƒ‚(:é îïêÔ¡Õœø disk.drbd1260.in‚0 disk.dm-0.in‚ 8763.32421875 disk.sda3.in‚0$ memory_percent‚99.95574951171875 network.recv‚ 1.263239e+06# disk.dm-0.out‚70759.82470703125 disk.sda.out‚ 107.890625 proc.memory.availableMB‚7.25 cpu‚25.298211931538212 proc.cpu.threads‚17( disk.nvme0c0n1.out‚70759.82470703125 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda.in‚0# proc.memory.rssMB‚ 2917.12890625* proc.memory.percent‚17.804741859436035% disk./.usageGB‚1174.1877250671387# disk.nvme0c0n1.in‚ 8763.32421875& disk.nvme0n1.out‚70759.82470703125 disk.dm-1.in‚0 disk.drbd1260.out‚0.515625 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 107.890625 network.sent‚ 5.172269e+06) disk./.usagePercent‚78.80072343837861! disk.nvme0n1.in‚ 8763.32421875 disk.dm-1.out‚ 107.890625Joi›„‚(: ýïêÔÛ­Þö gpu.0.gpu‚24.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚133.17' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.633999999999997 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0˜Èh3z…‚(:ï ýïêÔŽè¡ø disk.dm-0.in‚ 8763.32421875# disk.dm-0.out‚70924.50634765625 network.sent‚ 5.17823e+06% disk./.usageGB‚1174.0877532958984 disk.sda.out‚ 108.234375 disk.sda3.out‚ 108.234375# disk.nvme0c0n1.in‚ 8763.32421875( disk.nvme0c0n1.out‚70924.50634765625% proc.memory.availableMB‚ 7.2421875 cpu‚25.298088316414063 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.51953125$ memory_percent‚99.95579719543457* proc.memory.percent‚17.804741859436035 network.recv‚ 1.268068e+06) disk./.usagePercent‚78.79401424892878 disk.sda3.in‚0 disk.dm-1.out‚ 108.234375 disk.sda2.in‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 2917.12890625 proc.cpu.threads‚17 disk.sda.in‚0! disk.nvme0n1.in‚ 8763.32421875& disk.nvme0n1.out‚70924.50634765625 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.in‚06Aä_‚†‚(:÷ ŒðêÔíà÷ cpu‚25.297257438550314 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.51953125 disk.sda.out‚ 108.68359375# proc.memory.rssMB‚ 2917.12890625& disk.nvme0n1.out‚71005.47900390625 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 108.68359375 network.sent‚ 5.186379e+06 network.recv‚ 1.272578e+06# disk.nvme0c0n1.in‚ 8763.32421875! disk.nvme0n1.in‚ 8763.32421875) disk./.usagePercent‚78.79930157253344 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.dm-0.in‚ 8763.32421875# disk.dm-0.out‚71005.47900390625 disk.sda.in‚0$ memory_percent‚99.95419979095459& proc.memory.availableMB‚ 7.50390625% disk./.usageGB‚1174.1665382385254 disk.sda3.in‚0 disk.sda3.out‚ 108.68359375( disk.nvme0c0n1.out‚71005.47900390625 disk.sda2.out‚0* proc.memory.percent‚17.804741859436035Ž·ÚO‡‚(:… ŒðêÔ²Åé÷ gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚100.313' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.0626 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ëüù42ˆ- _wandbruntime‚3801  _runtime‚3801M=UljÁ trainer/global_step‚2249 % _timestamp‚1.7885245636697176e+09 _runtime‚3801.889356223 _step‚179 & lr-AdamW/pg1‚0.0008983239682632287 & lr-AdamW/pg2‚0.0008983239682632287û]jµ ŠÆ •×«uÃ_step‚179 & lr-AdamW/pg1‚0.0008983239682632287 & lr-AdamW/pg2‚0.0008983239682632287 trainer/global_step‚2249 % _timestamp‚1.7885245636697176e+09 _runtime‚3801.889356223³¿‹`~2‹- _wandbruntime‚3801  _runtime‚3801ÿgœ‹Î'ŒÈO F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 102859109 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚90305472518144 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚60758804 _step‚180 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚94753959122432 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4525878773760 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚94753959122432 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚42100305 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚3801.890721746 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚100245005394432 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42100307 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16006458 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚674 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4436136417369 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚90305472518144 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚41054968 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4435907436293 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚60785146 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚95718846486016 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚90503058508949 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚94754210792960 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚90305454515200 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 280134656 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚86067151072656 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4448738274816 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚94754210792960 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 102885453 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚41055644 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16005784 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚90503304529385 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚42100307 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚676 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚95718822545920 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚90305472518144 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚60758804 _step‚180 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚94753959122432 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4525878773760 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚94753959122432 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚42100305 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚3801.890721746 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚100245005394432 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42100307 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16006458 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚674 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4436136417369 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚90305472518144 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚41054968 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4435907436293´ƒ …2Ž- _wandbruntime‚3803  _runtime‚3803Ï@}íØ&ÒM OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4450468633088 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41072460 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25059988 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚100288693611008 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16012474 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310352896 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚86104681430080 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4450468633088 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚684 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚3803.820562716 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 102904076 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 102930446 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4437865925279 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4527876742144 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚42118582 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚682 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4528163154944 trainer/global_step‚2249 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚86104698469440 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚60785496 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚94795460869120 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 102904076 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚60811864 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚42118580 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16013156 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚95760530456064 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 102930446 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚90344758555136 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚42118582 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚41073144 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚60811864 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚90344758555136 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚90344740552192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚60785496 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286412800 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 % _timestamp‚1.7885245655971997e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚90344740552192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 î®/×M H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 102930446 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚90344758555136 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚42118582 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚41073144 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 NADevic4.p®eStatsMonitor.on_train_batch_end/active.small_pool.allocated‚60811864 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚90344758555136 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚90344740552192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚60785496 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286412800 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 % _timestamp‚1.7885245655971997e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚90344740552192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4450468633088 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41072460 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25059988 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚100288693611008 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16012474 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310352896 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚86104681430080 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4450468633088 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚684 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚3803.820562716 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 102904076 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 102930446 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4437865925279 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4527876742144 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚42118582 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚682 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4528163154944 trainer/global_step‚2249 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚86104698469440 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚60785496 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚94795460869120 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 102904076 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚60811864 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚42118580 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16013156 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚95760530456064 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0µu9¦2‘- _wandbruntime‚3803  _runtime‚3803÷?)X’Ò 0loss/FRt1/assignment_loss‚3.1792671697480337 2loss/SRqqt2/assignment_loss‚2.2462987122791156 1loss/FBt4/assignment_loss‚0.23745022396520815 1loss/SRqqt3/detection_loss‚1.0563045440210124 %loss/total_loss‚47.67554151118887 _step‚182 .loss/FRt4/detection_loss‚1.777108565349287 .loss/FBt1/detection_loss‚2.543592589742997 .loss/FBt3/detection_loss‚2.473300258318583 epoch‚1 % _timestamp‚1.7885245656047637e+09 .loss/FRt2/detection_loss‚3.252811441818873 0loss/FRt4/assignment_loss‚3.5297303944826126 0loss/FBt2/assignment_loss‚0.3572581110860725 .loss/FBt2/detection_loss‚2.383624545905901 trainer/global_step‚2249 /loss/FRt3/assignment_loss‚1.866288902128444 /loss/FRt1/detection_loss‚2.5640680977276396 1loss/SRqqt1/detection_loss‚1.8559790862103303 1loss/SRqqt4/assignment_loss‚2.316371420191394 0loss/SRqqt2/detection_loss‚2.598719088094575 /loss/FBt4/detection_loss‚1.2437185375859625 2loss/SRqqt3/assignment_loss‚1.0787104534742944 0loss/FBt1/assignment_loss‚0.2110213624250986 /loss/FRt2/assignment_loss‚4.665428042411804 2loss/SRqqt1/assignment_loss‚1.2396690127595018 2loss/FBt3/assignment_loss‚0.017627652002071652 .loss/FRt3/detection_loss‚1.873269344734795 1loss/SRqqt4/detection_loss‚3.1079239547252655 _runtime‚3803.824750297cÁ`Ï]“× 2loss/SRqqt3/assignment_loss‚1.0787104534742944 0loss/FBt1/assignment_loss‚0.2110213624250986 /loss/FRt2/assignment_loss‚4.665428042411804 2loss/SRqqt1/assignment_loss‚1.2396690127595018 2loss/FBt3/assignment_loss‚0.017627652002071652 .loss/FRt3/detection_loss‚1.873269344734795 1loss/SRqqt4/detection_loss‚3.1079239547252655 _runtime‚3803.824750297 0loss/FRt1/assignment_loss‚3.1792671697480337 2loss/SRqqt2/assignment_loss‚2.2462987122791156 1loss/FBt4/assignment_loss‚0.23745022396520815 1loss/SRqqt3/detection_loss‚1.0563045440210124 %loss/total_loss‚47.67554151118887 _step‚182 .loss/FRt4/detection_loss‚1.777108565349287 .loss/FBt1/detection_loss‚2.543592589742997 .loss/FBt3/detection_loss‚2.473300258318583 epoch‚1 % _timestamp‚1.7885245656047637e+09 .loss/FRt2/detection_loss‚3.252811441818873 0loss/FRt4/assignment_loss‚3.5297303944826126 0loss/FBt2/assignment_loss‚0.3572581110860725 .loss/FBt2/detection_loss‚2.383624545905901 trainer/global_step‚2249 /loss/FRt3/assignment_loss‚1.866288902128444 /loss/FRt1/detection_loss‚2.5640680977276396 1loss/SRqqt1/detection_loss‚1.8559790862103303 1loss/SRqqt4/assignment_loss‚2.316371420191394 0loss/SRqqt2/detection_loss‚2.598719088094575 /loss/FBt4/detection_loss‚1.2437185375859625¶ìðjœ”‚(:‘ ›ðêÔ üÏ÷ gpu.0.gpu‚13.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚101.433' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚20.286600000000004 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0˜¤~•‚(:ó ›ðêÔöž ø) disk./.usagePercent‚78.80301137771212 disk.dm-0.in‚ 8763.32421875# disk.dm-0.out‚71321.58740234375# disk.nvme0c0n1.in‚ 8763.32421875 disk.dm-1.in‚0 proc.cpu.threads‚17 disk.sda3.out‚ 109.02734375( disk.nvme0c0n1.out‚71321.58740234375& disk.nvme0n1.out‚71321.58740234375 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.in‚0# proc.memory.rssMB‚ 2917.12890625 cpu‚25.297002201251548% disk./.usageGB‚1174.2218170166016 disk.sda.in‚0! disk.nvme0n1.in‚ 8763.32421875 disk.dm-1.out‚ 109.02734375 disk.drbd1260.out‚ 0.52734375 disk.sda2.in‚0* proc.memory.percent‚17.804741859436035 network.recv‚ 1.275722e+06 disk.sda.out‚ 109.02734375 disk.sda3.in‚0 disk.sda2.out‚0$ memory_percent‚99.95613098144531" proc.memory.availableMB‚7.1875 network.sent‚ 5.189433e+06Ýaß3–‚(:… ªðêÔ—é›÷ gpu.0.gpu‚24.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚111.635' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.327 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÞoHñs—‚(:è ªðêÔÑ¢Éø$ memory_percent‚99.95627403259277# proc.memory.rssMB‚ 2917.12890625* proc.memory.percent‚17.804741859436035 cpu‚25.296816115724187 proc.cpu.threads‚17 network.recv‚ 1.281457e+06) disk./.usagePercent‚78.80459145621346$ disk./.usageGB‚1174.245361328125# disk.dm-0.out‚71508.65380859375 network.sent‚ 5.29728e+06 disk.sda1.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 109.53125 disk.drbd1260.out‚ 0.52734375 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda.out‚ 109.53125 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 109.53125" disk.nvme0c0n1.in‚ 8940.0234375( disk.nvme0c0n1.out‚71508.65380859375& disk.nvme0n1.out‚71508.65380859375% proc.memory.availableMB‚ 7.1640625 disk.dm-0.in‚ 8940.0234375 disk.sda.in‚0 disk.nvme0n1.in‚ 8940.0234375©j¿¾‘˜‚(:† ¹ðêÔäµ”ö gpu.0.gpu‚33.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚110.863' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.1726 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¤œWs™‚(:è ¹ðêÔ„›ýø network.recv‚ 1.285778e+06) disk./.usagePercent‚78.80490762551824 disk.sda2.out‚0 disk.sda1.out‚0 disk.dm-1.out‚ 109.859375 disk.drbd1260.out‚0.53125% proc.memory.availableMB‚ 7.1484375* proc.memory.percent‚17.804741859436035& disk.nvme0n1.out‚71514.13818359375 disk.sda1.in‚0 cpu‚25.296501667079752 proc.cpu.threads‚17 disk.sda3.out‚ 109.859375 network.sent‚ 5.30304e+06 disk.sda2.in‚0 disk.sda.in‚0 disk.sda.out‚ 109.859375 disk.sda3.in‚0 disk.drbd1260.in‚0# proc.memory.rssMB‚ 2917.12890625$ disk./.usageGB‚1174.250072479248 disk.dm-0.in‚ 8940.0234375# disk.dm-0.out‚71514.13818359375" disk.nvme0c0n1.in‚ 8940.0234375( disk.nvme0c0n1.out‚71514.13818359375 disk.nvme0n1.in‚ 8940.0234375 disk.dm-1.in‚0$ memory_percent‚99.95636940002441Ðصt‘š‚(:† ÈðêÔ©––ö gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚108.058' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.6116 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸™Ày›‚(:î ÈðêÔžŠ±ø network.sent‚ 5.309367e+06% disk./.usageGB‚1174.4266624450684 disk.dm-1.in‚0 disk.dm-1.out‚ 110.19921875# disk.dm-0.out‚71882.01318359375 disk.sda.in‚0 disk.sda3.out‚ 110.19921875( disk.nvme0c0n1.out‚71882.01318359375& disk.nvme0n1.out‚71882.01318359375 disk.drbd1260.in‚0 disk.drbd1260.out‚0.53125 disk.sda.out‚ 110.19921875# proc.memory.availableMB‚6.90625* proc.memory.percent‚17.804741859436035 proc.cpu.threads‚17) disk./.usagePercent‚78.81675872629329 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 8940.0234375# proc.memory.rssMB‚ 2917.12890625 cpu‚25.296395034901515 network.recv‚ 1.290559e+06 disk.nvme0n1.in‚ 8940.0234375 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.dm-0.in‚ 8940.0234375 disk.sda3.in‚0$ memory_percent‚99.957847595214840Ùp‘œ‚(:† ×ðêÔœÊâ÷ gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚136.678' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.3356 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÜPÌ8}‚(:ò ×ðêÔ±áœø disk.sda2.in‚0 disk.sda2.out‚0 disk.sda.out‚ 110.68359375 disk.sda3.in‚0 disk.sda3.out‚ 110.68359375( disk.nvme0c0n1.out‚72108.67822265625 disk.nvme0n1.in‚ 8940.0234375 disk.sda1.in‚0% disk./.usageGB‚1174.5734558105469 disk.sda.in‚0" disk.nvme0c0n1.in‚ 8940.0234375 disk.dm-1.out‚ 110.68359375# proc.memory.rssMB‚ 2917.12890625 cpu‚25.296283375181826 network.recv‚ 1.294066e+06( disk./.usagePercent‚78.8266101522184# disk.dm-0.out‚72108.67822265625& disk.nvme0n1.out‚72108.67822265625 disk.dm-1.in‚0 disk.drbd1260.in‚0$ memory_percent‚99.95636940002441% proc.memory.availableMB‚ 7.1484375 network.sent‚ 5.314393e+06 disk.dm-0.in‚ 8940.0234375 disk.sda1.out‚0 disk.drbd1260.out‚ 0.53515625* proc.memory.percent‚17.804741859436035 proc.cpu.threads‚17¾'Ÿj‘ž‚(:† æðêÔ≛ù gpu.0.gpu‚21.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚102.421' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.4842 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Œ {Ÿ‚(:ð æðêÔ˜×íþ disk.sda.out‚ 111.09765625# proc.memory.rssMB‚ 2917.12890625 proc.cpu.threads‚17 disk.sda3.out‚ 111.09765625( disk.nvme0c0n1.out‚72462.36572265625 network.recv‚ 1.297498e+06 disk.sda2.out‚0 disk.dm-0.in‚ 9103.765625 disk.sda.in‚0 disk.sda3.in‚0 disk.sda1.in‚0! disk.nvme0c0n1.in‚ 9103.765625& disk.nvme0n1.out‚72462.36572265625 disk.sda1.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 111.09765625 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.53515625$ memory_percent‚99.95636940002441 network.sent‚ 5.319643e+06) disk./.usagePercent‚78.84843761059525 disk.sda2.in‚0 disk.nvme0n1.in‚ 9103.765625% proc.memory.availableMB‚ 7.1484375* proc.memory.percent‚17.804741859436035 cpu‚25.296209228893648% disk./.usageGB‚1174.8987007141113# disk.dm-0.out‚72462.36572265625š2 - _wandbruntime‚3890  _runtime‚3890/[rhÇ¡Á & lr-AdamW/pg1‚0.0008963165316719868 & lr-AdamW/pg2‚0.0008963165316719868 trainer/global_step‚2299 % _timestamp‚1.7885246524209304e+09 _runtime‚3890.642613224 _step‚1833ü´8Ì¢Æ _step‚183 & lr-AdamW/pg1‚0.0008963165316719868 & lr-AdamW/pg2‚0.0008963165316719868 trainer/global_step‚2299 % _timestamp‚1.7885246524209304e+09 _runtime‚3890.642613224·kbL2£- _wandbruntime‚3890  _runtime‚3890¸¿þÎ'¤ÈO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4546953981440 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 105110811 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚97803044988416 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚712 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4533845476801 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚62123098 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚96816476678656 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚96816476678656 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4534074457877 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚92269774367744 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚41945069 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 _runtime‚3890.647315925 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚710 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚97803021048320 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4625689704448 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚41944357 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚92269774367744 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16355783 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 % _timestamp‚1.7885246524259403e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚92477514420657ö,Ò†Ç ¥ÍO XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚92477760441093 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚96816728349184 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16355073 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚102429033701888 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚87943685983216 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _step‚184 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚43014055 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚62096756 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 299009024 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚62123098 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4625988713472 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25589284 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚43014057 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚102428710752768 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚92269756364800 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4546720313856 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 trainer/global_step‚2299 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚9-Yàz 6816728349184 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4546720313856 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 322949120 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚43014055 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚62096756 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚92269774367744 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚41945069 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 _runtime‚3890.647315925 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚710 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚97803021048320 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4625689704448 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚41944357 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚92269774367744 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16355783 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 % _timestamp‚1.7885246524259403e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚92477514420657 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4546953981440 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 105110811 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚97803044988416 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚712 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4533845476801 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966899712 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚62123098 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚96816476678656 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚96816476678656 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4534074457877 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 trainer/global_step‚2299 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚92309042401792 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚41962831 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 105155778 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚92309042401792 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚97844728958464 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25600088 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚96857978425344 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16362071 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25600086 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 105182148 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚62149816 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41962157 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚87981199301280 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚62123448 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 $ _timestamp‚1.788524654890705e+09 _step‚185 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚96857978425344 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4627974446080 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚92309060404736 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43032332 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4548918020608 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43032332 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4548918020608 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 trainer/global_step‚2299 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚92309042401792 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚41962831 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 105155778 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚92309042401792 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚97844728958464 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25600088 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚96857978425344 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16362071 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565024768 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25600086 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 105182148 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚62149816 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41962157 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚87981199301280 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚62123448 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 $ _timestamp‚1.788524654890705e+09 _step‚185 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚96857978425344 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4627974446080 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚92309060404736 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 306171904 trainer/global_step‚2349 % _timestamp‚1.7885247433327734e+09 _runtime‚3981.555729979 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚63461050 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4645169688064 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 107388857 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚688 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚99887243490816 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 282231808 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚43927805 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 107362513 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚63461050 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚89820203854416 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚98878994234880 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4645169688064 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0̨çÓ'½ÍO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚94234076217344 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚43927807 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚63434708 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚98878994234880 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚94234058214400 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26129384 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981076 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚94234076217344 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚686 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 306171904 trainer/global_step‚2349 % _timestamp‚1.7885247433327734e+09 _runtime‚3981.555729979 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚63461050 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4645169688064 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 107388857 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚688 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚99887243490816 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 282231808 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚43927805 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26342 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020436 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 107362513 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚63461050 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚89820203854416 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚98878994234880 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4645169688064 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚98879245905408 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16704034 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16703348 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4632012498385 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4725779764736 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚42833420 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4631783517309 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚43927805 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚94234058214400 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚104613023255552 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26129386 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 107407480 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 _step‚189 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚63487768 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚94273362254336 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26140188 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4727781504512 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚94491459178775 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 107433850 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 294801408 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚89857717172480 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚94273362254336 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚98920244297728 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16710057 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4633742006295 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚98920495981568 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚98920495981568 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4647133727232 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 trainer/global_step‚2349 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4647133727232 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚94491705199419 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚94273344251392 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚700 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚104656390223872 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 107407480 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QBDevicõ¼c… eStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42850243 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚94273344251392 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚104656708965376 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26140186 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4646900046336 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚63487768 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43946082 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 107433850 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚43946080 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42850945 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247¸’fÝ&À×M E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚94273344251392 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚700 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚104656390223872 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 107407480 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42850243 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚94273344251392 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚104656708965376 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26140186 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4646900046336 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚63487768 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43946082 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 107433850 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚43946080 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42850945 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 _runtime‚3984.312768657 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43946080 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚63461400 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚99928903520768 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4727486703104 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26368 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚63461400 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 % _timestamp‚1.7885247460926945e+09 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4633970987579 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 107407480 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 _step‚189 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚63487768 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚94273362254336 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26140188 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4727781504512 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚94491459178775 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 107433850 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26370 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 294801408 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚89857717172480 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚94273362254336 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚98920244297728 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16710057 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4633742006295 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32247 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚98920495981568 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚98920495981568 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4647133727232 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 trainer/global_step‚2349 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4647133727232 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚94491705199419 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104½èl µ2Á- _wandbruntime‚3984  _runtime‚3984éÑõTÂÎ 0loss/FRt4/assignment_loss‚3.4646188439206873 0loss/FRt1/assignment_loss‚4.0892709526750775 1loss/FBt1/assignment_loss‚0.19353310092396392 1loss/FBt4/assignment_loss‚0.21317032728251561 epoch‚1 _runtime‚3984.314020245 /loss/FBt4/detection_loss‚1.3272207370609976 %loss/total_loss‚48.12223527891176 0loss/FRt2/assignment_loss‚3.7961094045639037 1loss/SRqqt4/assignment_loss‚2.072128818370402 0loss/FBt2/assignment_loss‚0.4263365222260274 /loss/FRt3/detection_loss‚1.5749724180652545 /loss/FRt4/detection_loss‚1.8454842880923064 1loss/SRqqt2/detection_loss‚2.6514308601617813 1loss/SRqqt1/assignment_loss‚1.232958019648989 2loss/SRqqt2/assignment_loss‚1.9545248374342918 1loss/FBt3/assignment_loss‚0.40237822863340017 0loss/FRt3/assignment_loss‚1.6025154709815979 0loss/SRqqt1/detection_loss‚2.266004885236422 -loss/FBt2/detection_loss‚2.60233798623085 .loss/FRt2/detection_loss‚3.187645733356476 _step‚190 .loss/SRqqt4/detection_loss‚3.0375413633883 /loss/FBt1/detection_loss‚2.5157820449935064 trainer/global_step‚2349 $ _timestamp‚1.788524746094582e+09 0loss/SRqqt3/assignment_loss‚1.09486144402808 1loss/SRqqt3/detection_loss‚0.9359877254785032 .loss/FRt1/detection_loss‚2.568238844474157 .loss/FBt3/detection_loss‚3.067182421684265Æ»fYÃÓ 0loss/FRt2/assignment_loss‚3.7961094045639037 1loss/SRqqt4/assignment_loss‚2.072128818370402 0loss/FBt2/assignment_loss‚0.4263365222260274 /loss/FRt3/detection_loss‚1.5749724180652545 /loss/FRt4/detection_loss‚1.8454842880923064 1loss/SRqqt2/detection_loss‚2.6514308601617813 1loss/SRqqt1/assignment_loss‚1.232958019648989 2loss/SRqqt2/assignment_loss‚1.9545248374342918 1loss/FBt3/assignment_loss‚0.40237822863340017 0loss/FRt3/assignment_loss‚1.6025154709815979 0loss/SRqqt1/detection_loss‚2.266004885236422 -loss/FBt2/detection_loss‚2.60233798623085 .loss/FRt2/detection_loss‚3.187645733356476 _step‚190 .loss/SRqqt4/detection_loss‚3.0375413633883 /loss/FBt1/detection_loss‚2.5157820449935064 trainer/global_step‚2349 $ _timestamp‚1.788524746094582e+09 0loss/SRqqt3/assignment_loss‚1.09486144402808 1loss/SRqqt3/detection_loss‚0.9359877254785032 .loss/FRt1/detection_loss‚2.568238844474157 .loss/FBt3/detection_loss‚3.067182421684265 0loss/FRt4/assignment_loss‚3.4646188439206873 0loss/FRt1/assignment_loss‚4.0892709526750775 1loss/FBt1/assignment_loss‚0.19353310092396392 1loss/FBt4/assignment_loss‚0.21317032728251561 epoch‚1 _runtime‚3984.314020245 /loss/FBt4/detection_loss‚1.3272207370609976 %loss/total_loss‚48.12223527891176¾ź—úÄ‚(:… ÏñêÔÞ€ïö gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚96.899' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.3798 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 EÆtÅ‚(:é ÏñêÔ̳ßú network.recv‚ 1.327455e+06 disk.sda2.out‚0 disk.dm-0.in‚ 9261.328125 disk.sda.in‚0 disk.drbd1260.out‚ 0.55078125 cpu‚25.29586600599673$ disk./.usageGB‚1175.638744354248 disk.sda3.in‚0( disk.nvme0c0n1.out‚73594.05126953125 disk.sda1.out‚0 disk.dm-1.in‚0& proc.memory.availableMB‚ 7.73828125# proc.memory.rssMB‚ 2917.12890625 network.sent‚ 5.460331e+06) disk./.usagePercent‚78.89810256022281 disk.sda2.in‚0 disk.sda.out‚ 114.234375 disk.sda1.in‚0 proc.cpu.threads‚17# disk.dm-0.out‚73594.05126953125 disk.sda3.out‚ 114.234375 disk.nvme0n1.in‚ 9261.328125& disk.nvme0n1.out‚73594.05126953125 disk.dm-1.out‚ 114.234375 disk.drbd1260.in‚0$ memory_percent‚99.95276927947998* proc.memory.percent‚17.804741859436035! disk.nvme0c0n1.in‚ 9261.328125ií ‘Æ‚(:† ÞñêÔÈúÞø gpu.0.gpu‚29.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚109.773' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.9546 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0s´*zÇ‚(:ï ÞñêÔ¿Þ¬ý) disk./.usagePercent‚78.89967751857345 disk.sda3.out‚ 114.7734375" disk.nvme0c0n1.in‚ 9402.8203125 disk.sda.out‚ 114.7734375 disk.sda1.out‚0 disk.dm-1.in‚0 disk.drbd1260.in‚0$ memory_percent‚99.95126724243164& disk.nvme0n1.out‚73935.25439453125 disk.dm-1.out‚ 114.7734375 disk.drbd1260.out‚ 0.55078125$ proc.memory.availableMB‚7.984375# proc.memory.rssMB‚ 2917.12890625 cpu‚25.295847620396565% disk./.usageGB‚1175.6622123718262( disk.nvme0c0n1.out‚73935.25439453125 disk.nvme0n1.in‚ 9402.8203125 disk.sda1.in‚0* proc.memory.percent‚17.804741859436035 proc.cpu.threads‚17 disk.sda3.in‚0 network.sent‚ 5.567781e+06 network.recv‚ 1.333038e+06 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 9402.8203125# disk.dm-0.out‚73935.25439453125 disk.sda.in‚0là¹zÈ‚(:„ íñêÔ²œâ÷ gpu.0.gpu‚6.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚92.852' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.5704 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚03† zÉ‚(:ï íñêÔú¬€ø network.recv‚ 1.337493e+06) disk./.usagePercent‚78.89967751857345 disk.drbd1260.out‚ 0.5546875( disk.nvme0c0n1.out‚73993.81982421875& disk.nvme0n1.out‚73993.81982421875# proc.memory.rssMB‚ 2917.12890625 cpu‚25.295686294581966 proc.cpu.threads‚17 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.sda.out‚ 115.2421875 disk.sda3.in‚0" disk.nvme0c0n1.in‚ 9402.8203125 disk.sda1.in‚0 disk.dm-1.in‚0$ memory_percent‚99.95131492614746 network.sent‚ 5.573625e+06 disk.sda2.out‚0 disk.dm-0.in‚ 9402.8203125 disk.nvme0n1.in‚ 9402.8203125 disk.dm-1.out‚ 115.2421875% proc.memory.availableMB‚ 7.9765625* proc.memory.percent‚17.804741859436035% disk./.usageGB‚1175.6622123718262# disk.dm-0.out‚73993.81982421875 disk.sda.in‚0 disk.sda3.out‚ 115.2421875 disk.sda1.out‚0‘IëïÊ‚(:‚ üñêÔâÍ•ø gpu.0.gpu‚21.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚103.6' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.72 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ò;b{Ë‚(:ð üñêÔþ“¯ù disk.sda1.in‚0 disk.sda3.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚78.89797609250088 disk.dm-1.in‚0 disk.dm-1.out‚ 115.54296875 disk.sda2.in‚0 disk.sda.in‚0 disk.sda3.out‚ 115.54296875# memory_percent‚99.9539852142334 disk.drbd1260.out‚ 0.5546875 network.recv‚ 1.342395e+06( disk.nvme0c0n1.out‚74142.63623046875 disk.nvme0n1.in‚ 9402.8203125& disk.nvme0n1.out‚74142.63623046875 disk.sda1.out‚0 disk.sda2.out‚0# disk.dm-0.out‚74142.63623046875" disk.nvme0c0n1.in‚ 9402.8203125 disk.dm-0.in‚ 9402.8203125 disk.sda.out‚ 115.54296875% proc.memory.availableMB‚ 7.5390625# proc.memory.rssMB‚ 2917.12890625* proc.memory.percent‚17.804741859436035 cpu‚25.29568454735844 disk.drbd1260.in‚0 network.sent‚ 5.579855e+06% disk./.usageGB‚1175.6368598937988e1Ç‘Ì‚(:† ‹òêÔùü™÷ gpu.0.gpu‚17.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚100.838' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.1676 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Pém~Í‚(:ó ‹òêÔÐÂæù disk.dm-1.out‚ 115.91796875 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 9402.8203125 network.recv‚ 1.345527e+06" disk.nvme0c0n1.in‚ 9402.8203125 disk.nvme0n1.in‚ 9402.8203125 disk.sda3.in‚0$ memory_percent‚99.95248317718506& proc.memory.availableMB‚ 7.78515625 network.sent‚ 5.585023e+06) disk./.usagePercent‚78.89955105085153$ disk./.usageGB‚1175.660327911377 disk.sda1.in‚0 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda3.out‚ 115.91796875& disk.nvme0n1.out‚74442.16552734375# disk.dm-0.out‚74442.16552734375 disk.sda.out‚ 115.91796875# proc.memory.rssMB‚ 2917.12890625* proc.memory.percent‚17.804741859436035 cpu‚25.295633842375906 proc.cpu.threads‚17 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.55859375( disk.nvme0c0n1.out‚74442.16552734375 disk.sda1.out‚0Àp–º~΂(:ó šòêԱ̖ø disk.dm-1.out‚ 116.42578125 disk.sda2.in‚0 disk.sda.in‚0 disk.sda3.out‚ 116.42578125# disk.nvme0c0n1.in‚ 9406.97265625' disk.nvme0c0n1.out‚74649.1162109375& proc.memory.availableMB‚ 7.53515625) disk./.usagePercent‚78.89955105085153 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.55859375 disk.sda2.out‚0 disk.dm-0.in‚ 9406.97265625% disk.nvme0n1.out‚74649.1162109375 disk.sda1.in‚0$ memory_percent‚99.95400905609131 network.recv‚ 1.349127e+06 disk.drbd1260.in‚0 disk.sda3.in‚0! disk.nvme0n1.in‚ 9406.97265625* proc.memory.percent‚17.804741859436035$ disk./.usageGB‚1175.660327911377" disk.dm-0.out‚74649.1162109375 disk.sda.out‚ 116.42578125 disk.sda1.out‚0# proc.memory.rssMB‚ 2917.12890625 cpu‚25.295731470941952 proc.cpu.threads‚17 network.sent‚ 5.590114e+06k}›Ï‚(: šòêÔØ„Öø gpu.0.gpu‚8.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚104.585' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚20.916999999999998 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ú5j2Ð- _wandbruntime‚4070  _runtime‚40709a:7ÄѾ % lr-AdamW/pg1‚0.000892028962911138 % lr-AdamW/pg2‚0.000892028962911138 trainer/global_step‚2399 $ _timestamp‚1.788524832207164e+09 _runtime‚4070.427046966 _step‚191ÓõÂ}ÉÒà % lr-AdamW/pg1‚0.000892028962911138 % lr-AdamW/pg2‚0.000892028962911138 trainer/global_step‚2399 $ _timestamp‚1.788524832207164e+09 _runtime‚4070.427046966 _step‚191¿ ¬ 2Ó- _wandbruntime‚4070  _runtime‚4070CrF•ì"ÔÎO VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚100941763461632 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚106796780204032 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚44841555 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17052119 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚106797096861696 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚101971441993216 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚4070.427963714 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚43721603 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 trainer/global_step‚2399 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44841557 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 109640559 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚96198378066944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4743151727104 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚91696721725616 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚101971418053120 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚100941511791104 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 109614215 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚44841555 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4743385394688 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚96426426244073 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚64772660 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 % _timestamp‚1.7885248322080083e+09 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚96198378066944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44841557 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 _step‚192 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚96198360064000 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26669484 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚91696704686256 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚96198360064000 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚64772660 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4825362150912 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚43722267 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 316657664 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4729950Gyýè538893 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚100941511791104 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26669486 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4743385394688 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292717568ØuÆïÙ'ÕÓO @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚64799002 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4729721557817 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17052781 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚100941763461632 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚96426672264509 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32004 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚91696721725616 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26344 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚101971418053120 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚100941511791104 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 109614215 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚44841555 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4743385394688 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚96426426244073 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚64772660 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233667584 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 % _timestamp‚1.7885248322080083e+09 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966899712 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚96198378066944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44841557 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 _step‚192 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚96198360064000 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26669484 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚91696704686256 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚96198360064000 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚64772660 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4825362150912 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚43722267 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 316657664 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4729950538893 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251670528 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565024768 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288769 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚100941511791104 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32247 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26669486 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059449 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4743385394688 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26342 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292717568 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚100941763461632 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚106796780204032 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32247 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚44841555 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32004 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17052119 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚106797096861696 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚101971441993216 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚4070.427963714 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565024768 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚43721603 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 trainer/global_step‚2399 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44841557 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26344 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 109640559 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233667584 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251670528 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚96198378066944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4743151727104 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚96237646100992 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26680288 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4731909028087 _step‚193 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44859832 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4745349433856 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4745349433856 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17059337 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 109685552 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4827643118592 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286412800 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚64799352 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚96466161111127 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 trainer/global_step‚2399 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17060022 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚102013102023168 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 109685552 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚102013125963264 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43740310 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 $ _timestamp‚1.788524833227163e+09 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚96465915090483 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 109659182 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26680286 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚100983013537792 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚44859830 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚100983013537792 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43739623 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44859830ç>Râ&ØÜM OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251683840 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44859830 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚106840458728960 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚91734252083040 _runtime‚4071.448249801 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚64825720 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966899712 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚96237646100992 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26680288 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4731909028087 _step‚193 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233680896 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32247 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981284 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288769 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44859832 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4745349433856 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4745349433856 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17059337 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 109685552 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4827643118592 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286412800 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚64799352 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚96466161111127 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 trainer/global_step‚2399 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26368 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17060022 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚102013102023168 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059449 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 109685552 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚102013125963264 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43740310 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020644 $ _timestamp‚1.788524833227163e+09 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚96465915090483 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 109659182 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26680286 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚100983013537792 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233680896 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565024768 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚44859830 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32004 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251683840 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966899712 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚100983013537792 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43739623 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0Á¢³êV2Ù- _wandbruntime‚4071  _runtime‚4071éDô~ëÚÓ 2loss/SRqqt3/assignment_loss‚1.1166071516651093 1loss/SRqqt2/detection_loss‚2.8662408998138025 0loss/SRqqt4/detection_loss‚2.941572436919579 _step‚194 /loss/FRt1/assignment_loss‚4.549602513727934 0loss/FRt2/assignment_loss‚3.1211614202369344 1loss/FBt1/assignment_loss‚0.35611758833911156 2loss/SRqqt2/assignment_loss‚1.7307915295425214 .loss/FRt4/detection_loss‚2.086141320088735 .loss/FBt1/detection_loss‚2.668407758076986 .loss/FBt2/detection_loss‚2.369513019271519 /loss/FRt3/assignment_loss‚2.204353551353727 0loss/FRt4/assignment_loss‚3.1075397626711774 2loss/SRqqt1/assignment_loss‚1.6974237244576216 1loss/FBt3/assignment_loss‚0.21688752617345536 .loss/FRt2/detection_loss‚3.405255588618192 % _timestamp‚1.7885248332297313e+09 _runtime‚4071.449243694 2loss/SRqqt4/assignment_loss‚1.4622260561356177 1loss/FBt4/assignment_loss‚0.21465092110408798 epoch‚1 0loss/FBt2/assignment_loss‚0.3771184832957527 /loss/FBt3/detection_loss‚2.3731801424707686 %loss/total_loss‚47.70004912562911 trainer/global_step‚2399 .loss/FRt1/detection_loss‚3.263754261576611 1loss/SRqqt1/detection_loss‚1.8886046301234851 1loss/SRqqt3/detection_loss‚1.024Žªân9114200725393 .loss/FBt4/detection_loss‚0.867469198954029 /loss/FRt3/detection_loss‚1.7905182209398065µt•^ÛØ /loss/FRt3/assignment_loss‚2.204353551353727 0loss/FRt4/assignment_loss‚3.1075397626711774 2loss/SRqqt1/assignment_loss‚1.6974237244576216 1loss/FBt3/assignment_loss‚0.21688752617345536 .loss/FRt2/detection_loss‚3.405255588618192 % _timestamp‚1.7885248332297313e+09 _runtime‚4071.449243694 2loss/SRqqt4/assignment_loss‚1.4622260561356177 1loss/FBt4/assignment_loss‚0.21465092110408798 epoch‚1 0loss/FBt2/assignment_loss‚0.3771184832957527 /loss/FBt3/detection_loss‚2.3731801424707686 %loss/total_loss‚47.70004912562911 trainer/global_step‚2399 .loss/FRt1/detection_loss‚3.263754261576611 1loss/SRqqt1/detection_loss‚1.8886046301234851 1loss/SRqqt3/detection_loss‚1.0249114200725393 .loss/FBt4/detection_loss‚0.867469198954029 /loss/FRt3/detection_loss‚1.7905182209398065 2loss/SRqqt3/assignment_loss‚1.1166071516651093 1loss/SRqqt2/detection_loss‚2.8662408998138025 0loss/SRqqt4/detection_loss‚2.941572436919579 _step‚194 /loss/FRt1/assignment_loss‚4.549602513727934 0loss/FRt2/assignment_loss‚3.1211614202369344 1loss/FBt1/assignment_loss‚0.35611758833911156 2loss/SRqqt2/assignment_loss‚1.7307915295425214 .loss/FRt4/detection_loss‚2.086141320088735 .loss/FBt1/detection_loss‚2.668407758076986 .loss/FBt2/detection_loss‚2.369513019271519ÂË›óq‘Ü‚(:† ©òêÔÒ•Ýõ gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚132.554' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.5108 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0iÅÁ¢YÝ‚(:Î ©òêԴЄø disk.drbd1260.in‚0 disk.sda2.in‚0! disk.dm-0.out‚74856.189453125 disk.sda.in‚0 disk.sda3.out‚117 disk.drbd1260.out‚0.5625 disk.sda2.out‚0 disk.sda.out‚117# proc.memory.rssMB‚ 2917.12890625* proc.memory.percent‚17.804741859436035 network.sent‚ 5.59597e+06! disk.nvme0n1.in‚ 9614.60546875$ disk.nvme0n1.out‚74856.189453125 disk.dm-1.out‚117 disk.dm-0.in‚ 9614.60546875 disk.sda3.in‚0$ memory_percent‚99.95098114013672 cpu‚25.295809094275732 network.recv‚ 1.353746e+06) disk./.usagePercent‚78.91330582365119# disk.nvme0c0n1.in‚ 9614.60546875& disk.nvme0c0n1.out‚74856.189453125 disk.sda1.in‚0 disk.sda1.out‚0# proc.memory.availableMB‚8.03125 proc.cpu.threads‚17% disk./.usageGB‚1175.8652839660645 disk.dm-1.in‚0¡œÞ‚(:‘ ¸òêÔº«‚÷ gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚123.037' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.607400000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0H°ýŸpß‚(:å ¸òêÔ«³ðù disk.drbd1260.out‚0.5625 disk.sda2.in‚0& proc.memory.availableMB‚ 7.78515625# proc.memory.rssMB‚ 2917.12890625 network.sent‚ 5.706574e+06 network.recv‚ 1.36054e+06 disk.sda1.out‚0 disk.dm-1.out‚ 117.453125 disk.sda2.out‚0 disk.sda.in‚0* proc.memory.percent‚17.804741859436035 cpu‚25.29562025070427# disk.nvme0c0n1.in‚ 9614.60546875$ disk.nvme0n1.out‚75197.361328125 disk.dm-1.in‚0! disk.dm-0.out‚75197.361328125 proc.cpu.threads‚17 disk.sda3.in‚0 disk.sda3.out‚ 117.453125& disk.nvme0c0n1.out‚75197.361328125 disk.sda1.in‚0 disk.dm-0.in‚ 9614.60546875 disk.sda.out‚ 117.453125$ memory_percent‚99.95248317718506) disk./.usagePercent‚78.91329814342517% disk./.usageGB‚1175.8651695251465! disk.nvme0n1.in‚ 9614.60546875 disk.drbd1260.in‚0­ÇYoà‚(:ä ÇòêԕLj÷% disk./.usageGB‚1175.8651504516602 disk.sda2.in‚0' disk.nvme0c0n1.out‚75255.8623046875! disk.nvme0n1.in‚ 9614.60546875% disk.nvme0n1.out‚75255.8623046875 disk.sda1.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 117.8515625 disk.sda3.out‚ 117.8515625# disk.nvme0c0n1.in‚ 9614.60546875 disk.sda1.out‚0$ memory_percent‚99.96185302734375* proc.memory.percent‚17.859935760498047 network.sent‚ 5.709729e+06 network.recv‚ 1.363768e+06" disk.dm-0.out‚75255.8623046875 disk.sda.out‚ 117.8515625 disk.drbd1260.in‚0! proc.memory.rssMB‚ 2926.171875 proc.cpu.threads‚17( disk./.usagePercent‚78.9132968633875 disk.sda2.out‚0 disk.dm-0.in‚ 9614.60546875 disk.sda.in‚0 disk.sda3.in‚0 disk.drbd1260.out‚0.5625 proc.memory.availableMB‚6.25 cpu‚25.299756742811958: ƒj‘á‚(:† ÇòêÔý†ëö gpu.0.gpu‚38.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚141.769' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.3538 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0>sÅâ‚(:„ ÖòêÔŒý¸û gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚96.612' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.3224 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0û!õ}|ã‚(:ñ ÖòêÔç´þ$ memory_percent‚99.97556209564209# disk.nvme0c0n1.in‚ 9614.60546875' disk.nvme0c0n1.out‚75685.3544921875 disk.drbd1260.out‚ 0.56640625 disk.sda2.out‚0 disk.sda.in‚0# proc.memory.rssMB‚ 2928.14453125 proc.cpu.threads‚17) disk./.usagePercent‚78.93164748344185! disk.nvme0n1.in‚ 9614.60546875% disk.nvme0n1.out‚75685.3544921875 disk.sda1.out‚0" disk.dm-0.out‚75685.3544921875 disk.sda.out‚ 118.1171875* proc.memory.percent‚17.871975898742676 cpu‚25.313197956488793 network.sent‚ 5.715042e+06 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda2.in‚0& proc.memory.availableMB‚ 4.00390625 network.recv‚ 1.367629e+06% disk./.usageGB‚1176.1385879516602 disk.sda1.in‚0 disk.dm-1.out‚ 118.1171875 disk.dm-0.in‚ 9614.60546875 disk.sda3.in‚0 disk.sda3.out‚ 118.1171875f2¬¶oä‚(:ä åòêÔ€éÀ÷# proc.memory.rssMB‚ 2927.14453125* proc.memory.percent‚17.865872383117676 network.recv‚ 1.372005e+06) disk./.usagePercent‚78.93164953150212% disk./.usageGB‚1176.1386184692383 disk.nvme0c0n1.in‚ 9744.71875 disk.nvme0n1.in‚ 9744.71875% disk.nvme0n1.out‚75972.0673828125 disk.sda1.in‚0 disk.dm-0.in‚ 9744.71875 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 118.578125 proc.cpu.threads‚17 network.sent‚ 5.720877e+06' disk.nvme0c0n1.out‚75972.0673828125 disk.drbd1260.out‚ 0.56640625$ memory_percent‚99.96960163116455& proc.memory.availableMB‚ 4.98046875 cpu‚25.32820570754411 disk.dm-1.in‚0 disk.dm-1.out‚ 118.578125 disk.drbd1260.in‚0 disk.sda2.in‚0" disk.dm-0.out‚75972.0673828125 disk.sda.out‚ 118.578125¯3Õ!å‚(:„ åòêÔ²îù gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚94.201' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.8402 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ïK¢äæ‚(:„ ôòêÔçͽø gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.031' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.6062 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ë}/mç‚(:â ôòêÔ®¹ñø network.recv‚ 1.376639e+06% disk./.usageGB‚1176.1151542663574 disk.nvme0n1.in‚ 9744.71875 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.5703125 disk.dm-0.in‚ 9744.71875" disk.dm-0.out‚76222.5595703125 network.sent‚ 5.727218e+06' disk./.usagePercent‚78.930074829159 disk.sda1.out‚0 disk.dm-1.out‚ 118.9609375 disk.sda.in‚0 disk.sda.out‚ 118.9609375 disk.nvme0c0n1.in‚ 9744.71875 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 118.9609375' disk.nvme0c0n1.out‚76222.5595703125" memory_percent‚99.961256980896 cpu‚25.339083505701495% disk.nvme0n1.out‚76222.5595703125 disk.sda1.in‚0 disk.sda2.in‚0& proc.memory.availableMB‚ 6.34765625" proc.memory.rssMB‚ 2925.8515625* proc.memory.percent‚17.857980728149414 proc.cpu.threads‚17¹ý‘á‘è‚(:† ƒóêÔ‘÷ƒû gpu.0.gpu‚22.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚105.192' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.0384 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0|:)Øré‚(:ç ƒóêÔñ“üþ disk.dm-0.in‚ 9744.71875 disk.sda.out‚ 119.44921875 disk.sda1.in‚0 disk.sda1.out‚0% proc.memory.availableMB‚ 30.078125* proc.memory.percent‚17.716026306152344 disk.sda.in‚0( disk.nvme0c0n1.out‚76770.40478515625 disk.nvme0n1.in‚ 9744.71875 disk.dm-1.out‚ 119.44921875 disk.drbd1260.out‚ 0.5703125 proc.memory.rssMB‚ 2902.59375 cpu‚25.35029928041224) disk./.usagePercent‚78.93007508516654 disk.sda2.in‚0 disk.sda2.out‚0# disk.dm-0.out‚76770.40478515625 disk.sda3.in‚0 disk.sda3.out‚ 119.44921875 disk.nvme0c0n1.in‚ 9744.71875 disk.drbd1260.in‚0% disk./.usageGB‚1176.1151580810547& disk.nvme0n1.out‚76770.40478515625 disk.dm-1.in‚0# memory_percent‚99.8164176940918 proc.cpu.threads‚17 network.sent‚ 5.732127e+06 network.recv‚ 1.380548e+06ÿßu|ê‚(:… ’óêÔášûõ gpu.0.gpu‚19.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚95.327' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.0654 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ƒ)”ñpë‚(:å ’óêÔ•Žæ÷ network.recv‚ 1.383866e+06 disk.nvme0n1.in‚ 9744.71875 disk.drbd1260.out‚ 0.57421875 disk.dm-0.in‚ 9744.71875# disk.dm-0.out‚77020.96826171875 disk.sda.in‚0 proc.memory.rssMB‚ 2902.59375* proc.memory.percent‚17.716026306152344% disk./.usageGB‚1176.0713157653809( disk.nvme0c0n1.out‚77020.96826171875& disk.nvme0n1.out‚77020.96826171875 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 network.sent‚ 5.737253e+06) disk./.usagePercent‚78.92713279057551 disk.nvme0c0n1.in‚ 9744.71875 disk.sda1.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 119.765625 disk.sda.out‚ 119.765625& proc.memory.availableMB‚ 29.8359375 disk.sda2.in‚0 disk.sda3.out‚ 119.765625$ memory_percent‚99.81789588928223 cpu‚25.362898679905154 proc.cpu.threads‚17A{ö–2ì- _wandbruntime‚4192  _runtime‚4192®RÊW;*íµT UHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚45942500 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚27447142 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚98418012804608 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚27447140 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚309 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 4803938232832 YLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚65742512 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚104251139410432 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 188859824 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 4803938232832 `MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚98418012804608 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 4886839617024 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 151344640 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 4803762421760 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚103221775226368 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 6966899712 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 _step‚195 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚20709 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚109138002967552 `MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚93662204309280 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 572522496 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚382 ^PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 111664301 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚582 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 6966899712 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚98418030807552 UHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚45942498 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚582 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚104251163350528 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 4886688272384 YFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚98452350958011 VGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 7134511104 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚20711 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚20709 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 4803762421760 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚109137827682816 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 6741059449 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 560288769 SDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 7134511104 UHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚65742512 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 6458321920 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚309 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚103221969040384 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚17459760 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚17459378 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 339145216 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚44906518 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 6236507136 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚273 YLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚45942500 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚32004 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 572522496 QDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚65721803 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚103221775226368 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚32247 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚309 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚309 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 4803938232832 YLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚65742512 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚104251139410432 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 188859824 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 4803938232832 `MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚98418012804608 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 4886839617024 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 151344640 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 4803762421760 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚103221775226368 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 6966899712 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 _step‚195 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚20709 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚109138002967552 `MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚93662204309280 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 572522496 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚382 ^PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 111664301 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚582 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 6966899712 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚98418030807552 UHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚45942498 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚582 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚104251163350528 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 4886688272384 YFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚98452350958011 VGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 7134511104 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚20711 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚20709 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 4803762421760 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚109137827682816 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 6741059449 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 560288769 SDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 7134511104 UHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚65742512 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 6458321920 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚309 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚103221969040384 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚17459760 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚17459378 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 339145216 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚44906518 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 6236507136 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚273 YLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚45942500 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚32004 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 572522496 QDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚65721803 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚103221775226368 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚32247 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚309 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚582 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20711 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 4804412092928 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚17464968 WDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚98478382815455 VGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 7134511104 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚273 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚103252314760704 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚104281501164032 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 188859824 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚309 bODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚98448096481792 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚20709 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 6741059449 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 111688591 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚582 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚45957882 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚32004 RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 6966899712 XJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 565024768 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚44923790 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚309 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 111709302 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚103252508574720 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 4804236281856 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32247 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32247 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚582 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20711 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 4804412092928 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚17464968 WDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚98478382815455 VGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 7134511104 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚273 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚103252314760704 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚104281501164032 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 188859824 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚309 bODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚98448096481792 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚20709 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 6741059449 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 111688591Ä«‰ª&‘ò‚(:† ¡óêÔ·ú›÷ gpu.0.gpu‚19.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚113.954' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.7908 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0{ù³Ó€ó‚(:õ ¡óêÔýÚÆù disk.sda3.in‚0 disk.drbd1260.in‚0% proc.memory.availableMB‚ 22.828125# proc.memory.rssMB‚ 2909.91796875* proc.memory.percent‚17.760729789733887 proc.cpu.threads‚17 network.sent‚ 5.743096e+06 network.recv‚ 1.388535e+06$ disk./.usageGB‚1175.262493133545 disk.sda2.out‚0 disk.sda.in‚0# disk.nvme0c0n1.in‚ 9922.48046875( disk.nvme0c0n1.out‚77169.83935546875& disk.nvme0n1.out‚77169.83935546875) disk./.usagePercent‚78.87285202510562 disk.sda2.in‚0# disk.dm-0.out‚77169.83935546875 disk.sda.out‚ 120.26171875 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0$ memory_percent‚99.86066818237305 disk.sda3.out‚ 120.26171875! disk.nvme0n1.in‚ 9922.48046875 disk.dm-1.out‚ 120.26171875 disk.drbd1260.out‚ 0.57421875 cpu‚25.379120849016346 disk.dm-0.in‚ 9922.48046875jªÚZ2ô- _wandbruntime‚4209  _runtime‚4209¥ÀXD¾$õœ¥ ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 -jet/accuracy_4_of_5‚0.0076804920099675655 particle/accuracy_1_of_1‚1 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 &particle/f_score‚0.369279146194458 jet/accuracy_4_of_6‚0 particle/accuracy_5_of_7‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.2935185432434082 +"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.75 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.9166666865348816 % _timestamp‚1.7885249711913488e+09 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 +jet/accuracy_4_of_4‚0.00664961151778698 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 particle/accuracy_1_of_6‚1 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3473542332649231 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5357142686843872 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3333333432674408 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.13960854709148407 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.09802237153053284 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.16494254767894745 9"Purity/0FRt*SRqqt3FBt/event_purity‚0.1315789520740509 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.14929793775081635 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 particle/accuracy_6_of_6‚1 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.46666666865348816 & Purity/1FRt*SRqqt4FBt/FBt_purity‚1 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.24626436829566956 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4982757866382599 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.34310340881347656 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_4_of_7‚1 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ( Purity/4FRt1SRqqt0FBt/FRt_purity‚0.5 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9987770915031433 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 jet/accuracy_6_of_7‚0 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.015287797898054123 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9939283132553101 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9801075458526611 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.026002658531069756 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.24697734415531158 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 *jet/accuracy_2_of_4‚0.3379571735858917 ) Purity/4FRt*SRqqt0FBt/FRt_purity‚0.25 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4642244279384613 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5526315569877625 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.016511207446455956 particle/accuracy_2_of_6‚1 particle/accuracy_4_of_6‚1 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.3333333432674408 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.20149904489517212 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5463991761207581 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 jet/accuracy_5_of_7‚0 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.2777777910232544 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_7‚1 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.11925782263278961 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.06899762898683548 particle/accuracy_2_of_5‚1 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.21875 particle/accuracy_3_of_7‚1 ) Purity/3FRt3SRqqt0FBt/FRt_purity‚0.25 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 ,jet/accuracy_3_of_3‚0.019462430849671364 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.3435356616973877 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 jet/accuracy_7_of_7‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.1614493727684021 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.375 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.15211895108222961 *jet/accuracy_2_of_5‚0.4599398076534271 jet/accuracy_1_of_6‚1 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9980175495147705 'particle/accuracy‚0.841223418712616 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.4791666865348816 8"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.519413948059082 particle/accuracy_1_of_2‚1 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.09848485141992569 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.6875 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7222222089767456 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9873519539833069 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.3279278874397278 6 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.949999988079071 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4603980779647827 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.14267677068710327 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.28333333134651184 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.4782562851905823 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5044866800308228 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.14785344898700714 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_7‚1 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.43670764565467834 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.46894755959510803 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.44919946789741516 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.006540063768625259 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 ,&PuĘËåmrity/1FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.14559386670589447 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.4791666865348816 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5720664262771606 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 particle/accuracy_2_of_2‚1 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.12841108441352844 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2731974422931671 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8167262077331543 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1288013458251953 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.3408123850822449 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.10226611793041229 8"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.501118540763855 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.15311986207962036 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6163689494132996 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.24377688765525818 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_2_of_3‚0.22870458662509918 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4406309425830841 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.31705012917518616 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.33424267172813416 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6363636255264282 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.6515151262283325 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ."Purity/0FRt0SRqqt3FBt/event_purity‚0.15625 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 jet/accuracy_4_of_7‚0 particle/accuracy_7_of_7‚0.5 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_3_of_6‚1 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 & Purity/*FRt1SRqqt4FBt/FBt_purity‚1 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.49739745259284973 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.12841209769248962 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.06666666269302368 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5285639762878418 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9267140030860901 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9942509531974792 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.988645613193512 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.3396514058113098 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.02753818780183792 *jet/accuracy_1_of_2‚0.7346689105033875 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚0 _runtime‚4209.414737979 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9957482814788818 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4072829782962799 +particle/sensitivity‚0.2978479862213135 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.1666666716337204 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.02625300921499729 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.22914116084575653 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9837301969528198 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.10849057137966156 6 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.949999988079071 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 trainer/global_step‚2417 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.2041097730398178 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.46894755959510803 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5116079449653625 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4930555522441864 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 6 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.144404798746109 ) Purity/3FRt3SRqqt*FBt/FRt_purity‚0.25 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 6 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.356080025434494 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.13742682337760925 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2291666716337204 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.2142857164144516 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.3986704647541046 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5687801241874695 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.9166666865348816 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.06936520338058472 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.46873316168785095 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.12154297530651093 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9191052317619324 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 +particle/specificity‚0.9417788982391357 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.34375 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 epoch‚1 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.07004371285438538 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5827874541282654 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.1666666716337204 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 _step‚197 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.17110803723335266 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5704022645950317 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.19259260594844818 *"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.5166666507720947 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9932336211204529 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9891926646232605 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.03807471692562103 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.4748532176017761 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4857456386089325 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.03124680370092392 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.15285420417785645 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1706721931695938 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9259259700775146 particle/accuracy_2_of_3‚1 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.35227271914482117 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.3333333432674408 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 & Purity/*FRt1SRqqt3FBt/FRt_purity‚0 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.18093110620975494 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.028004415333271027 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.1112465187907219 jet/accuracy_3_of_7‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9908046126365662 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 jet/accuracy_5_of_6‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1547619104385376 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.3783244490623474 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.44816383719444275 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.12832264602184296 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.11316872388124466 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚0 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.75 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.3912207782268524 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 :"Purity/2FRt*SRqqt2FBt/event_purity‚0.04651162773370743 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 +jet/accuracy_3_of_6‚0.08974359929561615 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.3499999940395355 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.3826046884059906 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9989093542098999 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2474597692489624 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4903791546821594 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5350794792175293 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.1876247227191925 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.15895064175128937 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3729718029499054 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 +"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.25 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.375 9"Purity/*FRt1SRqqt3FBt/event_purity‚0.1111111119389534 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚0 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3706676959991455 8"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.421686589717865 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.47670748829841614 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.8998519778251648 particle/accuracy_4_of_4‚1 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 8"Purity/0FRt0SRqqt2FBt/event_purity‚0.350425660610199 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9963098764419556 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9861111640930176 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.05639880895614624 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4436398148536682 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 particle/accuracy_6_of_7‚1 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9946674704551697 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9923497438430786 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.013820321299135685 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_4‚1 particle/accuracy_4_of_5‚1 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6774832010269165 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9960132241249084 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 & Purity/1FRt1SRqqt4FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.13475176692008972 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.31928062438964844 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.45802953839302063 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3467261791229248 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.43333330750465393 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2708333134651184 *loss/val_total_loss‚47.270975025112406 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.07734721899032593 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.18547290563583374 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5119978785514832 ;"Purity/1FRt1SRqqt2FBt/event_purity‚0.013888888992369175 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12967544794082642 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.13770514726638794 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.1666666716337204 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.24372979998588562 6 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.953703761100769 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.34220269322395325 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.3499999940395355 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.625 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.6470588445663452 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.15311986207962036 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 6validation_average_jet_accuracy‚0.4206138253211975 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.33653196692466736 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚0 6 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.262297660112381 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5555555820465088 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.15380115807056427 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.75 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9365079998970032 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.20801609754562378 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 jet/accuracy_1_of_7‚1 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.4768988788127899 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 9"Purity/0FRt1SRqqt3FBt/event_purity‚0.1428571492433548 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5160399675369263 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9925234317779541 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9899424910545349 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.08311624825000763 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.010321661829948425 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3463541567325592 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3541666567325592 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_3_of_4‚0.04926956817507744 particle/accuracy_5_of_6‚1 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1301395148038864 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5444092154502869 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.992708683013916 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9914650917053223 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5014045834541321 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6363955736160278 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 particle/accuracy_1_of_5‚1 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6842105388641357 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5458679795265198 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.3333333134651184 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ) Purity/4FRt*SRqqt*FBt/FRt_purity‚0.25 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.44684597849845886 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4575669467449188 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6372460722923279 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9166667461395264 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8569892048835754 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 ( Purity/4FRt1SRqqt*FBt/FRt_purity‚0.5 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.14319345355033875 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4850388169288635 ,"Purity/1FRt0SRqqt3FBt/event_purity‚0.125 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.4166666567325592 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2053571492433548 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9047619104385376 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.38055554032325745 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.4292635917663574 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 5 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.99358731508255 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9919511079788208 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.09109792858362198 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.06578947603702545 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4521666169166565 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.40967804193496704 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5228987336158752 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.08830004930496216 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1111111119389534 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.4956844747066498 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9322916865348816 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.2380952537059784 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2363707274198532 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.013907785527408123 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5817900896072388 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.5714285969734192 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5408881902694702 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5274034738540649 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.3404139280319214 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.030932899564504623 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9920728802680969 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.989021897315979 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5585537552833557 jet/accuracy_2_of_7‚0 6 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.250444233417511 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5150473713874817 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.15203826129436493 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9987664818763733 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.65625 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5024306178092957 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.18097557127475739 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1932467818260193 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.01720263808965683 *jet/accuracy_1_of_1‚0.4663976728916168 jet/accuracy_5_of_5‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.6329364776611328 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 +jet/accuracy_2_of_6‚0.43589746952056885 jet/accuracy_6_of_6‚0 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9853801131248474 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2361111044883728 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9642857313156128 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 & Purity/1FRt1SRqqt3FBt/FRt_purity‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.4883648753166199¿Çþ ö¡¥ particle/accuracy_1_of_2‚1 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.09848485141992569 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.6875 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7222222089767456 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9873519539833069 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.3279278874397278 6 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.949999988079071 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4603980779647827 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.14267677068710327 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.28333333134651184 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.4782562851905823 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5044866800308228 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.14785344898700714 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_7‚1 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.43670764565467834 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.46894755959510803 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.44919946789741516 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.006540063768625259 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.14559386670589447 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.4791666865348816 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5720664262771606 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 particle/accuracy_2_of_2‚1 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.12841108441352844 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2731974422931671 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8167262077331543 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1288013458251953 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.3408123850822449 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.10226611793041229 8"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.501118540763855 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.15311986207962036 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6163689494132996 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.24377688765525818 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_2_of_3‚0.22870458662509918 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4406309425830841 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.31705012917518616 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.33424267172813416 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6363636255264282 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.6515151262283325 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ."Purity/0FRt0SRqqt3FBt/event_purity‚0.15625 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 jet/accuracy_4_of_7‚0 particle/accuracy_7_of_7‚0.5 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_3_of_6‚1 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 & Purity/*FRt1SRqqt4FBt/FBt_purity‚1 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0º_ãˆù.49739745259284973 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.12841209769248962 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.06666666269302368 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5285639762878418 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9267140030860901 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9942509531974792 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.988645613193512 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.3396514058113098 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.02753818780183792 *jet/accuracy_1_of_2‚0.7346689105033875 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚0 _runtime‚4209.414737979 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9957482814788818 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4072829782962799 +particle/sensitivity‚0.2978479862213135 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.1666666716337204 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.02625300921499729 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.22914116084575653 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9837301969528198 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.10849057137966156 6 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.949999988079071 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 trainer/global_step‚2417 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.2041097730398178 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.46894755959510803 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5116079449653625 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4930555522441864 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 6 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.144404798746109 ) Purity/3FRt3SRqqt*FBt/FRt_purity‚0.25 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 6 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.356080025434494 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.13742682337760925 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2291666716337204 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.2142857164144516 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.3986704647541046 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5687801241874695 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.9166666865348816 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.06936520338058472 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.46873316168785095 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.12154297530651093 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9191052317619324 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 +particle/specificity‚0.9417788982391357 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.34375 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 epoch‚1 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.07004371285438538 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5827874541282654 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.1666666716337204 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 _step‚197 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.17110803723335266 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5704022645950317 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.19259260594844818 *"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.5166666507720947 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9932336211204529 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9891926646232605 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.03807471692562103 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.4748532176017761 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4857456386089325 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.03124680370092392 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.15285420417785645 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1706721931695938 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9259259700775146 particle/accuracy_2_of_3‚1 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.35227271914482117 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.3333333432674408 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 & Purity/*FRt1SRqqt3FBt/FRt_purity‚0 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.18093110620975494 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.028004415333271027 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.1112465187907219 jet/accuracy_3_of_7‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9908046126365662 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 jet/accuracy_5_of_6‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1547619104385376 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.3783244490623474 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.44816383719444275 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.12832264602184296 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.11316872388124466 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚0 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.75 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.3912207782268524 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 :"Purity/2FRt*SRqqt2FBt/event_purity‚0.04651162773370743 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 +jet/accuracy_3_of_6‚0.08974359929561615 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.3499999940395355 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.3826046884059906 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9989093542098999 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2474597692489624 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4903791546821594 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5350794792175293 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.1876247227191925 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.15895064175128937 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3729718029499054 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 +"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.25 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.375 9"Purity/*FRt1SRqqt3FBt/event_purity‚0.1111111119389534 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚0 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3706676959991455 8"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.421686589717865 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.47670748829841614 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.8998519778251648 particle/accuracy_4_of_4‚1 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 8"Purity/0FRt0SRqqt2FBt/event_purity‚0.350425660610199 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9963098764419556 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9861111640930176 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.05639880895614624 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4436398148536682 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 particle/accuracy_6_of_7‚1 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9946674704551697 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9923497438430786 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.013820321299135685 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_4‚1 particle/accuracy_4_of_5‚1 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6774832010269165 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9960132241249084 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 & Purity/1FRt1SRqqt4FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.13475176692008972 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.31928062438964844 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.45802953839302063 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3467261791229248 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.43333330750465393 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2708333134651184 *loss/val_total_loss‚47.270975025112406 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.07734721899032593 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.18547290563583374 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5119978785514832 ;"Purity/1FRt1SRqqt2FBt/event_purity‚0.013888888992369175 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12967544794082642 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.13770514726638794 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.1666666716337204 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.24372979998588562 6 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.953703761100769 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.34220269322395325 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.3499999940395355 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.625 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.6470588445663452 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.15311986207962036 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 6validation_average_jet_accuracy‚0.4206138253211975 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.33653196692466736 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚0 6 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.262297660112381 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5555555820465088 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.15380115807056427 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.75 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9365079998970032 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.20801609754562378 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 jet/accuracy_1_of_7‚1 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.4768988788127899 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 9"Purity/0FRt1SRqqt3FBt/event_purity‚0.1428571492433548 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5160399675369263 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9925234317779541 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9899424910545349 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.08311624825000763 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.010321661829948425 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3463541567325592 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3541666567325592 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_3_of_4‚0.04926956817507744 particle/accuracy_5_of_6‚1 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1301395148038864 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5444092154502869 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.992708683013916 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9914650917053223 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5014045834541321 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6363955736160278 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 particle/accuracy_1_of_5‚1 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6842105388641357 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5458679795265198 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.3333333134651184 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ) Purity/4FRt*SRqqt*FBt/FRt_purity‚0.25 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.44684597849845886 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4575669467449188 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6372460722923279 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9166667461395264 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8569892048835754 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 ( Purity/4FRt1SRqqt*FBt/FRt_purity‚0.5 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.14319345355033875 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4850388169288635 ,"Purity/1FRt0SRqqt3FBt/event_purity‚0.125 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.4166666567325592 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2053571492433548 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9047619104385376 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.38055554032325745 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.4292635917663574 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 5 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.99358731508255 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9919511079788208 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.09109792858362198 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.06578947603702545 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4521666169166565 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.40967804193496704 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5228987336158752 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.08830004930496216 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1111111119389534 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.4956844747066498 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9322916865348816 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.2380952537059784 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2363707274198532 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.013907785527408123 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5817900896072388 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.5714285969734192 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5408881902694702 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5274034738540649 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.3404139280319214 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.030932899564504623 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9920728802680969 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.989021897315979 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5585537552833557 jet/accuracy_2_of_7‚0 6 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.250444233417511 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5150473713874817 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.15203826129436493 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9987664818763733 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.65625 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5024306178092957 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.18097557127475739 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1932467818260193 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.01720263808965683 *jet/accuracy_1_of_1‚0.4663976728916168 jet/accuracy_5_of_5‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.6329364776611328 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 +jet/accuracy_2_of_6‚0.43589746952056885 jet/accuracy_6_of_6‚0 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9853801131248474 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2361111044883728 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9642857313156128 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 & Purity/1FRt1SRqqt3FBt/FRt_purity‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.4883648753166199 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 -jet/accuracy_4_of_5‚0.0076804920099675655 particle/accuracy_1_of_1‚1 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 &particle/f_score‚0.369279146194458 jet/accuracy_4_of_6‚0 particle/accuracy_5_of_7‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.2935185432434082 +"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.75 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.9166666865348816 % _timestamp‚1.7885249711913488e+09 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 +jet/accuracy_4_of_4‚0.00664961151778698 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 particle/accuracy_1_of_6‚1 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3473542332649231 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5357142686843872 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3333333432674408 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.13960854709148407 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.09802237153053284 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.16494254767894745 9"Purity/0FRt*SRqqt3FBt/event_purity‚0.1315789520740509 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.14929793775081635 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 particle/accuracy_6_of_6‚1 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.46666666865348816 & Purity/1FRt*SRqqt4FBt/FBt_purity‚1 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.24626436829566956 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4982757866382599 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.34310340881347656 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_4_of_7‚1 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ( Purity/4FRt1SRqqt0FBt/FRt_purity‚0.5 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9987770915031433 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 jet/accuracy_6_of_7‚0 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.015287797898054123 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9939283132553101 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9801075458526611 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.026002658531069756 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.24697734415531158 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 *jet/accuracy_2_of_4‚0.3379571735858917 ) Purity/4FRt*SRqqt0FBt/FRt_purity‚0.25 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4642244279384613 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5526315569877625 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.016511207446455956 particle/accuracy_2_of_6‚1 particle/accuracy_4_of_6‚1 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.3333333432674408 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.20149904489517212 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5463991761207581 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 jet/accuracy_5_of_7‚0 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.2777777910232544 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_7‚1 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.11925782263278961 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.06899762898683548 particle/accuracy_2_of_5‚1 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.21875 particle/accuracy_3_of_7‚1 ) Purity/3FRt3SRqqt0FBt/FRt_purity‚0.25 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 ,jet/accuracy_3_of_3‚0.019462430849671364 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.3435356616973877 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 jet/accuracy_7_of_7‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.1614493727684021 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.375 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.15211895108222961 *jet/accuracy_2_of_5‚0.4599398076534271 jet/accuracy_1_of_6‚1 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9980175495147705 »)±¢'particle/accuracy‚0.841223418712616 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.4791666865348816 8"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.519413948059082ÅôC.‘÷‚(:† °óêÔÅêù gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚126.547' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.3094 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0uÀô7pø‚(:å °óêÔñ¥Þû proc.cpu.threads‚17 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda2.in‚0# disk.nvme0n1.out‚77803.18359375 disk.dm-0.out‚77803.18359375 disk.sda.in‚0 disk.sda.out‚ 120.73046875 disk.sda3.in‚0 disk.sda3.out‚ 120.73046875! disk.nvme0n1.in‚ 9922.48046875$ memory_percent‚97.88360595703125 proc.memory.rssMB‚ 2926.59375 network.recv‚ 1.394977e+06 disk.sda1.in‚0 disk.drbd1260.out‚0.578125 disk.dm-0.in‚ 9922.48046875# disk.nvme0c0n1.in‚ 9922.48046875% disk.nvme0c0n1.out‚77803.18359375* proc.memory.percent‚17.862510681152344 cpu‚25.388953920839402 network.sent‚ 5.851328e+06) disk./.usagePercent‚78.88240853035127% disk./.usageGB‚1175.4048919677734 disk.sda1.out‚0 disk.dm-1.out‚ 120.73046875 disk.sda2.out‚0" proc.memory.availableMB‚346.75›{¢"fù‚(:Û ¿óêÔü¤ì÷ disk.sda2.out‚0 disk.dm-0.in‚ 9922.48046875 disk.sda3.in‚0# disk.nvme0c0n1.in‚ 9922.48046875 disk.sda1.out‚0 proc.memory.rssMB‚ 2926.59375 cpu‚25.387814116572105 proc.cpu.threads‚17% disk./.usageGB‚1175.4276809692383 disk.sda.out‚121.1875 disk.sda3.out‚121.1875! disk.nvme0n1.in‚ 9922.48046875 disk.sda1.in‚0 disk.dm-1.in‚0 disk.drbd1260.out‚0.578125* proc.memory.percent‚17.862510681152344 network.sent‚ 5.974034e+06 network.recv‚ 1.400635e+06 disk.sda2.in‚0 disk.sda.in‚0$ disk.nvme0c0n1.out‚ 78256.9921875" disk.nvme0n1.out‚ 78256.9921875 disk.drbd1260.in‚0$ memory_percent‚97.88508415222168) disk./.usagePercent‚78.88393791936082 disk.dm-0.out‚ 78256.9921875 disk.dm-1.out‚121.1875' proc.memory.availableMB‚ 346.5078125î‹ŽP‘ú‚(:† ¿óêÔÀšŸù gpu.0.gpu‚30.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚33.0 gpu.0.powerWatts‚136.711' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.3422 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0…1‘û‚(:† ÎóêÔõ§Êõ gpu.0.gpu‚29.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚107.771' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.5542 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™7_ü‚(:ô ÎóêÔìò¤û) disk./.usagePercent‚78.90078961530219 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.58203125 disk.dm-0.in‚ 9922.48046875 disk.sda.out‚ 121.56640625 disk.sda3.in‚0# disk.nvme0c0n1.in‚ 9922.48046875 cpu‚25.386709132394312 disk.sda2.in‚0 disk.sda2.out‚0# disk.dm-0.out‚78425.82470703125 disk.sda.in‚0 disk.sda3.out‚ 121.56640625( disk.nvme0c0n1.out‚78425.82470703125& disk.nvme0n1.out‚78425.82470703125$ memory_percent‚97.88050651550293 network.sent‚ 5.979165e+06 network.recv‚ 1.404238e+06$ disk./.usageGB‚1175.678783416748 disk.sda1.out‚0' proc.memory.availableMB‚ 347.2578125 proc.cpu.threads‚17! disk.nvme0n1.in‚ 9922.48046875 disk.sda1.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 121.56640625 proc.memory.rssMB‚ 2925.59375* proc.memory.percent‚17.856407165527344hDÐî2ý- _wandbruntime‚4246  _runtime‚4246úáøÆþÀ & lr-AdamW/pg1‚0.0008897497242644344 & lr-AdamW/pg2‚0.0008897497242644344 trainer/global_step‚2449 $ _timestamp‚1.788525008644896e+09 _runtime‚4246.864642139 _step‚198·äøüËÿÅ $ _timestamp‚1.788525008644896e+09 _runtime‚4246.864642139 _step‚198 & lr-AdamW/pg1‚0.0008897497242644344 & lr-AdamW/pg2‚0.0008897497242644344 trainer/global_step‚2449ÆÇEðÐ2€- _wandbruntime‚4246  _runtime‚4246 #Â+Ú'ÔO F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 113398119 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17747800 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚95183497523632 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268544 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚66700091 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4955296561664 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚66673748 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46724371 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 113424464 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚104928036998656 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚667 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚105968407032320 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚66700091 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27945552 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚100056817393664 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚95183480484272 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚100040906282123 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚46724371 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚45693350 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17748467 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4871471276032 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4857654778931 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚105968430972416 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284328448 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 _step‚199 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4857425797851 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚100056799390720 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚66673748 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453}Ì=ß'‚ÙO 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚66673748 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46724371 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 113424464 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚104928036998656 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚667 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚105968407032320 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚66700091 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27945552 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚100056817393664 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚95183480484272 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚100040906282123 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚46724371 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚45693350 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17748467 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4871471276032 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4857654778931 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚105968430972416 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284328448 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 _step‚199 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4857425797851 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚100056799390720 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚66673748 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 113398119 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17747800 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚95183497523632 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268544 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚66700091 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4955296561664ÇÓT)d2ƒ- _wandbruntime‚4248  _runtime‚4248çI£: „ÞM H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚106010091002368 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4873201633792 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚46742646 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚66700440 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27956352 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4873435315200 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚66726809 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27956354 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4957578075136 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 113469457 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚200 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚100096085427712 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚104969538745856 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45711558 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚639 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 308255232 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚100080641149181 % _timestamp‚1.7885250097821302e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4859384286837 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45710919 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚637 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚106010114942464 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚100096103430656 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚100096085427712 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 _runtime‚4248.003153338 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚100096103430656 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚66700440 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4859613268125 trainer/global_step‚2449 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 113469457 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚46742648 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚95221027881056 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 113443086 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608ÎNÉ÷é&…ãM J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚66726809 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 284315136 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4859384286837 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45710919 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚637 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚106010114942464 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚100096103430656 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚100096085427712 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 _runtime‚4248.003153338 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚100096103430656 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚66700440 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4859613268125 trainer/global_step‚2449 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 113469457 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚46742648 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚95221027881056 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 113443086 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚106010091002368 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4873201633792 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚46742646 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚66700440 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27956352 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4873435315200 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚66726809 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27956354 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4957578075136 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 113469457 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚200 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚100096085427712 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚104969538745856 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45711558 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚639 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 308255232 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚100080641149181 % _timestamp‚1.7885250097821302e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4969453314560 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚102021101240320 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚106990806225920 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚106990554554880 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚68011700 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 _runtime‚4320.183597641 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚102021119243264 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28485652 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 115649821 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚106990554554880 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚102015608214271 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 316657152 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚46579487 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 trainer/global_step‚2499 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚732 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚102015362193831 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _step‚203 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292717056 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18093103 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚47638123 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚113107769108992 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚108052605534720 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚68038043 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚68011700 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18093835 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚2734§ ²˜ÙO JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18093103 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚47638123 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚113107769108992 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚108052605534720 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚68038043 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚68011700 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18093835 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4969453314560 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚102021101240320 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚106990806225920 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚106990554554880 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚68011700 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 _runtime‚4320.183597641 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚102021119243264 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28485652 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 115649821 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚106990554554880 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚102015608214271 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 316657152 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚46579487 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 trainer/global_step‚2499 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚732 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚102015362193831 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _step‚203 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292717056Ë@Xý™2™- _wandbruntime‚4322  _runtime‚4322ÕüEä&šÞM \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4971651021824 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚113151446605312 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4971417340416 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚102060405280256 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28496452 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18100433 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚2499 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚107031804617728 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚102054851040241 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 115721159 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 115694788 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚47656396 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚660 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4957551308633 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚658 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚97097528712896 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚107032056302080 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18101091 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 115694788 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46596885 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47656396 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4971417340416 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚68064761 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚113151756957696 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7885250840895898e+09 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚68064761 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚108094313444864 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4957322327345 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚68038392 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚68038392 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286412288 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 115721159 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚108094289504768 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 _runtime‚4322.311585579 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚46597545†x 6é&›ãM NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚68064761 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚108094313444864 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4957322327345 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚68038392 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚68038392 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286412288 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 115721159 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚108094289504768 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 _runtime‚4322.311585579 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚46597545 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4971651021824 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚113151446605312 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4971417340416 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚102060405280256 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28496452 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18100433 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚2499 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚107031804617728 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚102054851040241 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 115721159 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 115694788 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚47656396 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚660 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4957551308633 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚658 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚97097528712896 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚107032056302080 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18101091 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 115694788 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46596885 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47656396 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4971417340416 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚68064761 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚113151756957696 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7885250840895898e+09 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0ÌÜÁVF2œ- _wandbruntime‚4322  _runtime‚4322QwIIVÐ .loss/FRt3/detection_loss‚1.757493572174639 trainer/global_step‚2499 2loss/SRqqt2/assignment_loss‚3.3181540173079287 0loss/FBt4/assignment_loss‚0.2526652910684555 .loss/FRt2/detection_loss‚3.111313951512178 /loss/FBt2/detection_loss‚2.5426934186150048 _step‚205 .loss/FBt1/detection_loss‚2.104883364655755 /loss/FRt2/assignment_loss‚4.419423257311185 0loss/FBt1/assignment_loss‚0.3586959689442169 0loss/SRqqt2/detection_loss‚2.891609299750555 0loss/SRqqt3/detection_loss‚1.076811337426048 .loss/FBt3/detection_loss‚4.750773638486862 _runtime‚4322.315713766 %loss/total_loss‚51.35508419619913 % _timestamp‚1.7885250840954137e+09 /loss/FRt1/assignment_loss‚3.584026729518717 /loss/FRt4/assignment_loss‚3.184332463824958 2loss/SRqqt3/assignment_loss‚1.3476934049535414 2loss/SRqqt4/assignment_loss‚1.8205969663602966 /loss/FRt4/detection_loss‚1.8046311553436936 0loss/SRqqt4/detection_loss‚2.829651721886226 0loss/FRt3/assignment_loss‚1.9724042268766873 1loss/FBt2/assignment_loss‚0.36376667142843977 0loss/FBt3/assignment_loss‚0.3338714181445539 1loss/SRqqt1/detection_loss‚1.9613566921978463 /loss/FBt4/detection_loss‚0.9451340250146221 epoch‚2 2loss/SRqqt1/assignment_loss‚1.6283692939252388 /loss/FRt1/detection_loss‚2.9947323094714773EÃ[žÕ 1loss/SRqqt1/detection_loss‚1.9613566921978463 /loss/FBt4/detection_loss‚0.9451340250146221 epoch‚2 2loss/SRqqt1/assignment_loss‚1.6283692939252388 /loss/FRt1/detection_loss‚2.9947323094714773 .loss/FRt3/detection_loss‚1.757493572174639 trainer/global_step‚2499 2loss/SRqqt2/assignment_loss‚3.3181540173079287 0loss/FBt4/assignment_loss‚0.2526652910684555 .loss/FRt2/detection_loss‚3.111313951512178 /loss/FBt2/detection_loss‚2.5426934186150048 _step‚205 .loss/FBt1/detection_loss‚2.104883364655755 /loss/FRt2/assignment_loss‚4.419423257311185 0loss/FBt1/assignment_loss‚0.3586959689442169 0loss/SRqqt2/detection_loss‚2.891609299750555 0loss/SRqqt3/detection_loss‚1.076811337426048 .loss/FBt3/detection_loss‚4.750773638486862 _runtime‚4322.315713766 %loss/total_loss‚51.35508419619913 % _timestamp‚1.7885250840954137e+09 /loss/FRt1/assignment_loss‚3.584026729518717 /loss/FRt4/assignment_loss‚3.184332463824958 2loss/SRqqt3/assignment_loss‚1.3476934049535414 2loss/SRqqt4/assignment_loss‚1.8205969663602966 /loss/FRt4/detection_loss‚1.8046311553436936 0loss/SRqqt4/detection_loss‚2.829651721886226 0loss/FRt3/assignment_loss‚1.9724042268766873 1loss/FBt2/assignment_loss‚0.36376667142843977 0loss/FBt3/assignment_loss‚0.3338714181445539Í”¯z»¹Ÿ‚(:… ¨ôêÔ¡þÍ÷ gpu.0.gpu‚14.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.p›!Þ×owerWatts‚93.871' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.7742 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¹Wéè~ ‚(:ó ¨ôêÔƒÏÚø disk.sda3.in‚0 network.sent‚ 6.115802e+06 disk.drbd1260.in‚0 disk.sda2.out‚0 disk.dm-0.in‚10192.76953125# disk.dm-0.out‚80277.66943359375 disk.sda.in‚0$ disk.nvme0c0n1.in‚10192.76953125% disk./.usageGB‚1176.2431030273438 disk.sda3.out‚ 124.26953125( disk.nvme0c0n1.out‚80277.66943359375 disk.dm-1.out‚ 124.26953125* proc.memory.percent‚17.850303649902344 proc.cpu.threads‚17 disk.sda.out‚ 124.26953125 network.recv‚ 1.432113e+06 disk.drbd1260.out‚0.59375$ memory_percent‚98.23617935180664& proc.memory.availableMB‚ 288.984375 proc.memory.rssMB‚ 2924.59375 cpu‚25.38250963509209) disk./.usagePercent‚78.93866157786495 disk.sda2.in‚0" disk.nvme0n1.in‚10192.76953125& disk.nvme0n1.out‚80277.66943359375 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0ëRþ?¡‚(:„ ·ôêÔÇ£±ù gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚92.813' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.5626 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"ü¾y¢‚(:î ·ôêÔ ‰°û network.sent‚ 6.223105e+06) disk./.usagePercent‚78.93854176633893 disk.dm-1.out‚ 124.703125 disk.sda2.in‚0# disk.dm-0.out‚80368.79638671875 disk.sda.in‚0* proc.memory.percent‚17.850303649902344 proc.cpu.threads‚17 disk.drbd1260.out‚0.59375( disk.nvme0c0n1.out‚80368.79638671875 disk.sda.out‚ 124.703125' proc.memory.availableMB‚ 175.0390625 cpu‚25.382139551551315& disk.nvme0n1.out‚80368.79638671875 disk.drbd1260.in‚0 disk.dm-0.in‚10192.76953125 disk.sda3.in‚0 disk.sda3.out‚ 124.703125# memory_percent‚98.9316463470459 proc.memory.rssMB‚ 2924.59375 network.recv‚ 1.437716e+06% disk./.usageGB‚1176.2413177490234$ disk.nvme0c0n1.in‚10192.76953125" disk.nvme0n1.in‚10192.76953125 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0 disk.sda2.out‚0Ât~〣‚(:õ ÆôêÔË‚÷÷# disk.dm-0.out‚80524.19482421875 disk.sda.in‚0 disk.sda.out‚ 124.93359375$ memory_percent‚98.93016815185547% proc.memory.availableMB‚ 175.28125% disk./.usageGB‚1176.3037567138672$ disk.nvme0c0n1.in‚10192.76953125 disk.sda1.out‚0 disk.sda2.in‚0 disk.dm-0.in‚10192.76953125 proc.memory.rssMB‚ 2924.59375 cpu‚25.38184525148627 network.sent‚ 6.228311e+06 disk.sda3.in‚0 disk.sda3.out‚ 124.93359375* proc.memory.percent‚17.850303649902344 proc.cpu.threads‚17 network.recv‚ 1.441537e+06) disk./.usagePercent‚78.94273209765937( disk.nvme0c0n1.out‚80524.19482421875& disk.nvme0n1.out‚80524.19482421875 disk.sda1.in‚0 disk.dm-1.out‚ 124.93359375 disk.drbd1260.out‚ 0.59765625 disk.sda2.out‚0" disk.nvme0n1.in‚10192.76953125 disk.dm-1.in‚0 disk.drbd1260.in‚0?vt‘¤‚(:† ÆôêÔ»îæø gpu.0.gpu‚19.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚102.956' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.5912 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0îž‹ß‘¥‚(:† ÕôêÔ¦óû gpu.0.gpu‚24.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚102.782' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.5564 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ÇrŸ~¦‚(:ó ÕôêÔ«éàý disk.drbd1260.in‚0# disk.dm-0.out‚80941.51513671875$ memory_percent‚98.93321990966797% proc.memory.availableMB‚ 174.78125 proc.cpu.threads‚17 network.recv‚ 1.445756e+06 disk.sda1.in‚0 disk.drbd1260.out‚ 0.59765625 disk.sda.out‚ 125.30859375 disk.sda3.in‚0 disk.sda3.out‚ 125.30859375 cpu‚25.38144279255552 network.sent‚ 6.23416e+06 disk.dm-1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 proc.memory.rssMB‚ 2924.59375) disk./.usagePercent‚78.94613802189491$ disk.nvme0c0n1.in‚10192.76953125" disk.nvme0n1.in‚10192.76953125& disk.nvme0n1.out‚80941.51513671875 disk.dm-1.out‚ 125.30859375 disk.dm-0.in‚10192.76953125 disk.sda.in‚0* proc.memory.percent‚17.850303649902344$ disk./.usageGB‚1176.354507446289( disk.nvme0c0n1.out‚80941.51513671875 disk.sda1.out‚0,]v§‚(:ë äôêÔø¥Œû network.sent‚ 6.242312e+06 disk.dm-1.in‚0 disk.dm-1.out‚ 125.8984375 disk.sda.out‚ 125.8984375 disk.sda3.out‚ 125.8984375 disk.sda1.out‚0( proc.memory.availableMB‚ 175.27734375* proc.memory.percent‚17.850303649902344% disk./.usageGB‚1176.4912567138672 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.6015625 disk.sda2.out‚0 disk.dm-0.in‚ 10325.9453125 proc.memory.rssMB‚ 2924.59375 cpu‚25.38103050698226 proc.cpu.threads‚17 disk.sda.in‚0$ disk.nvme0n1.out‚81336.654296875 disk.sda1.in‚0$ memory_percent‚98.93019199371338! disk.dm-0.out‚81336.654296875 network.recv‚ 1.450617e+06) disk./.usagePercent‚78.95531537998237 disk.sda2.in‚0 disk.sda3.in‚0# disk.nvme0c0n1.in‚ 10325.9453125& disk.nvme0c0n1.out‚81336.654296875! disk.nvme0n1.in‚ 10325.9453125?Ë Ëœ¨‚(:‘ äôêÔ¤«ˆû gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚128.955' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.791000000000004 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ü^¨2©- _wandbruntime‚4406  _runtime‚4406ɷǪÁ _step‚206 & lr-AdamW/pg1‚0.0008849227132890706 & lr-AdamW/pg2‚0.0008849227132890706 trainer/global_step‚2549 % _timestamp‚1.7885251682895453e+09 _runtime‚4406.510041633%°Š¥Ì«Æ _step‚206 & lr-AdamW/pg1‚0.0008849227132890706 & lr-AdamW/pg2‚0.0008849227132890706 trainer/global_step‚2549 % _timestamp‚1.7885251682895453e+09 _runtime‚4406.510041633Î`ÇË2¬- _wandbruntime‚4406  _runtime‚4406’×È)Ú'­ÔO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚69375995 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚48551873 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚109053072111104 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚47465418 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚110136804037120 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 117927868 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18440360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5053530859947 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚103985403089920 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚69375995 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚103985421092864 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29025752 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚98936533266032 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 trainer/global_step‚2549 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚692 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚4406.513613704 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29025750 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 _step‚207 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚48551871 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚69349652 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18439668 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚98936516226672 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚103990064125979 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5155394906624 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚47466112 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462848 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522752 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 % _timestamp‚1.7885251682927232e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5067669021184 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5067669021184 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚115291910420992 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚110136827977216 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚69349652 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚109053072111104 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚103989818105539 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚103985421092864 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚109053323782144 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚00üÜwß'®ÙO H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚47466112 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462848 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522752 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 % _timestamp‚1.7885251682927232e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5067669021184 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5067669021184 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚115291910420992 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚110136827977216 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚69349652 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚109053072111104 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚103989818105539 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚103985421092864 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚109053323782144 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚69375995 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚48551873 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚109053072111104 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚47465418 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚110136804037120 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 117927868 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18440360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5053530859947 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚103985403089920 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚69375995 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚103985421092864 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29025752 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚98936533266032 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 trainer/global_step‚2549 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚692 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚4406.513613704 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29025750 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 _step‚207 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚48551871 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚69349652 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18439668 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚98936516226672 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚103990064125979 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5155394906624 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944ÏtÖÚ2¯- _wandbruntime‚4408  _runtime‚4408ЃûQ°ÞM SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚69402713 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚48570148 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 117972861 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚69376344 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 _runtime‚4408.336345482 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 % _timestamp‚1.7885251701142979e+09 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29036552 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚98974046584096 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚104024707129856 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5069866728448 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚109094573858304 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5157393418752 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 117972861 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚98974063623456 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5055260367853 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚711 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47483380 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚48570146 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚109094322173952 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚2549 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚104029306951949 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚48570148 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚115335905366016 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚115335586625024 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚47484093 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5157098617856 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 117946490 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚69402713 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5069633047040 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 117946490 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚109094573858304 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚104024689126912 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 _step‚208 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5069633047040 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚104029552972597 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚104024707129856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚69376344 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29036554 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18446828 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚110178488007168 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚104024689126912 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 294800896 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18447539èùïé&±ãM M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29036552 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚98974046584096 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚104024707129856 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5069866728448 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚109094573858304 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5157393418752 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 117972861 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚98974063623456 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5055260367853 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚711 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47483380 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚48570146 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚109094322173952 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚2549 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚104029306951949 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚48570148 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚115335905366016 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚115335586625024 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚47484093 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5157098617856 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 117946490 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚69402713 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5069633047040 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 117946490 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚109094573858304 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚104024689126912 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 _step‚208 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5069633047040 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚104029552972597 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚104024707129856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚69376344 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29036554 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18446828 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚110178488007168 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚104024689126912 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 294800896 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18447539 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚69402713 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚48570148 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 117972861 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚69376344 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 _runtime‚4408.336345482 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 % _timestamp‚1.7885251701142979e+09Ћþ)™2²- _wandbruntime‚4408  _runtime‚4408Ÿ@/W³Ñ /loss/FRt4/assignment_loss‚3.099907939080839 epoch‚2 _step‚209 2loss/SRqqt3/assignment_loss‚1.1345919941981824 1loss/SRqqt4/assignment_loss‚2.050859896838665 1loss/FBt3/assignment_loss‚0.12883495349867896 .loss/FRt2/detection_loss‚3.302249866983165 0loss/SRqqt4/detection_loss‚2.636749557654063 1loss/SRqqt1/assignment_loss‚1.994383505028155 1loss/SRqqt3/detection_loss‚1.1305147145803158 -loss/FRt3/detection_loss‚1.79415395285093 /loss/FRt4/detection_loss‚1.7313607418564734 1loss/SRqqt2/detection_loss‚2.8544968234168158 /loss/FBt4/detection_loss‚0.7231081206708738 0loss/FBt1/assignment_loss‚0.2736720716756491 1loss/FBt2/assignment_loss‚0.05218686280635542 .loss/FBt2/detection_loss‚2.696250855922699 $ _timestamp‚1.788525170118345e+09 /loss/FBt3/detection_loss‚2.8302467928992376 &loss/total_loss‚48.761214591982636 /loss/FRt2/assignment_loss‚4.055929090665734 1loss/FBt4/assignment_loss‚0.16055507476930683 1loss/SRqqt1/detection_loss‚2.4086827950345144 trainer/global_step‚2549 /loss/FRt1/assignment_loss‚4.379547547210347 2loss/SRqqt2/assignment_loss‚1.9538341225297362 .loss/FRt1/detection_loss‚2.656051202253862 /loss/FBt1/detection_loss‚2.5486940030868235 _runtime‚4408.338217589 0loss/FRt3/assignment_loss‚2.1643521064712155»_<—\´Ö &loss/total_loss‚48.761214591982636 /loss/FRt2/assignment_loss‚4.055929090665734 1loss/FBt4/assignment_loss‚0.16055507476930683 1loss/SRqqt1/detection_loss‚2.4086827950345144 trainer/global_step‚2549 /loss/FRt1/assignment_loss‚4.379547547210347 2loss/SRqqt2/assignment_loss‚1.9538341225297362 .loss/FRt1/detection_loss‚2.656051202253862 /loss/FBt1/detection_loss‚2.5486940030868235 _runtime‚4408.338217589 0loss/FRt3/assignment_loss‚2.1643521064712155 /loss/FRt4/assignment_loss‚3.099907939080839 epoch‚2 _step‚209 2loss/SRqqt3/assignment_loss‚1.1345919941981824 1loss/SRqqt4/assignment_loss‚2.050859896838665 1loss/FBt3/assignment_loss‚0.12883495349867896 .loss/FRt2/detection_loss‚3.302249866983165 0loss/SRqqt4/detection_loss‚2.636749557654063 1loss/SRqqt1/assignment_loss‚1.994383505028155 1loss/SRqqt3/detection_loss‚1.1305147145803158 -loss/FRt3/detection_loss‚1.79415395285093 /loss/FRt4/detection_loss‚1.7313607418564734 1loss/SRqqt2/detection_loss‚2.8544968234168158 /loss/FBt4/detection_loss‚0.7231081206708738 0loss/FBt1/assignment_loss‚0.2736720716756491 1loss/FBt2/assignment_loss‚0.05218686280635542 .loss/FBt2/detection_loss‚2.696250855922699 $ _timestamp‚1.788525170118345e+09 /loss/FBt3/detection_loss‚2.8302467928992376Ñ]Y ,›µ‚(: óôêÔ¥ÀŒø gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚133.17' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.633999999999997 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Æžht¶‚(:é óôêÔ‘бø disk.drbd1260.in‚0 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 10325.9453125$ memory_percent‚98.93016815185547% proc.memory.availableMB‚ 175.28125 proc.memory.rssMB‚ 2924.59375 cpu‚25.380678351911452 network.sent‚ 6.245467e+06% disk./.usageGB‚1176.4914894104004 disk.dm-1.out‚ 126.1640625 disk.sda.in‚0& disk.nvme0c0n1.out‚81629.650390625! disk.nvme0n1.in‚ 10325.9453125 disk.sda1.in‚0* proc.memory.percent‚17.850303649902344 network.recv‚ 1.454234e+06 disk.sda2.out‚0! disk.dm-0.out‚81629.650390625 disk.sda3.in‚0$ disk.nvme0n1.out‚81629.650390625 proc.cpu.threads‚17 disk.sda.out‚ 126.1640625) disk./.usagePercent‚78.95533099644194 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.6015625 disk.dm-0.in‚ 10325.9453125 disk.sda3.out‚ 126.1640625 disk.sda1.out‚0L=¦7‘·‚(:† ‚õêÔ‚ËÞö gpu.0.gpu‚14.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚116.356' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.2712 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ñ7ª»z¸‚(:ï ‚õêÔë‹õ÷ disk.sda1.out‚0 disk.dm-1.out‚ 126.515625) disk./.usagePercent‚78.97210384206277 disk.dm-0.in‚ 10325.9453125( disk.nvme0c0n1.out‚81932.64892578125! disk.nvme0n1.in‚ 10325.9453125 disk.sda1.in‚0$ memory_percent‚98.93105030059814 proc.memory.rssMB‚ 2924.59375* proc.memory.percent‚17.850303649902344 network.sent‚ 6.353019e+06 network.recv‚ 1.459822e+06 disk.sda2.out‚0 disk.sda.in‚0 disk.sda.out‚ 126.515625 disk.dm-1.in‚0 disk.drbd1260.in‚0( proc.memory.availableMB‚ 175.13671875% disk./.usageGB‚1176.7414169311523 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 10325.9453125& disk.nvme0n1.out‚81932.64892578125 disk.drbd1260.out‚ 0.60546875 cpu‚25.38030156208698 proc.cpu.threads‚17# disk.dm-0.out‚81932.64892578125 disk.sda3.in‚0 disk.sda3.out‚ 126.515625eûT\‘¹‚(:† ‘õêÔúçø gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚115.966' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.1932 gpu.0.smClock‚1305 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0÷V’°º‚(:ô ‘õêÔÉî«ø network.recv‚ 1.464215e+06 disk.dm-1.in‚0 disk.dm-1.out‚ 126.87890625 disk.sda3.out‚ 126.87890625# disk.nvme0c0n1.in‚ 10325.9453125( disk.nvme0c0n1.out‚82153.61376953125! disk.nvme0n1.in‚ 10325.9453125 cpu‚25.379861404072237 network.sent‚ 6.359123e+06 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.60546875 disk.dm-0.in‚ 10325.9453125 disk.sda.out‚ 126.87890625& disk.nvme0n1.out‚82153.61376953125* proc.memory.percent‚17.850303649902344) disk./.usagePercent‚78.97052888371212 disk.sda2.in‚0# disk.dm-0.out‚82153.61376953125 disk.sda.in‚0 disk.sda3.in‚0$ memory_percent‚98.93255233764648% disk./.usageGB‚1176.7179489135742 disk.sda1.in‚0 disk.sda2.out‚0& proc.memory.availableMB‚ 174.890625 proc.memory.rssMB‚ 2924.59375 proc.cpu.threads‚17 ùåo»‚(:…  õêÔúÄÄ÷ gpu.0.gpu‚3.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚102.662' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.5324 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ù䬼‚(:ô  õêÔâ²²ø% disk./.usageGB‚1177.7454872131348 disk.sda2.in‚0& disk.nvme0n1.out‚82392.28076171875$ memory_percent‚98.93107414245605* proc.memory.percent‚17.850303649902344 proc.cpu.threads‚17 network.recv‚ 1.46928e+06) disk./.usagePercent‚79.03948784115768# disk.dm-0.out‚82392.28076171875 disk.sda.in‚0 disk.sda3.out‚ 127.48046875$ disk.nvme0c0n1.in‚10517.17578125( disk.nvme0c0n1.out‚82392.28076171875" disk.nvme0n1.in‚10517.17578125 network.sent‚ 6.365206e+06 disk.drbd1260.out‚0.609375 disk.sda2.out‚0 disk.dm-0.in‚10517.17578125 disk.sda1.in‚0 disk.dm-1.out‚ 127.48046875' proc.memory.availableMB‚ 175.1328125 cpu‚25.37948397216919 disk.drbd1260.in‚0 disk.sda.out‚ 127.48046875 disk.sda3.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0 proc.memory.rssMB‚ 2924.59375¦Üص½‚(:… ¯õêÔœ³ÿø gpu.0.gpu‚24.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚103.195' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.639 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆD•’¾‚(:ô ¯õêÔß“™ý network.sent‚ 6.370281e+06 disk.sda3.out‚ 128.1796875$ disk.nvme0c0n1.in‚10517.17578125 disk.dm-1.out‚ 128.1796875 proc.memory.rssMB‚ 2924.59375 disk.sda2.out‚0# disk.dm-0.out‚82869.63623046875 disk.sda.in‚0 disk.sda.out‚ 128.1796875 disk.sda1.in‚0 disk.dm-1.in‚0 disk.drbd1260.in‚0( proc.memory.availableMB‚ 175.12890625 network.recv‚ 1.472769e+06) disk./.usagePercent‚79.07290706467741% disk./.usageGB‚1178.2434577941895 disk.sda2.in‚0( disk.nvme0c0n1.out‚82869.63623046875& disk.nvme0n1.out‚82869.63623046875* proc.memory.percent‚17.850303649902344 cpu‚25.379219517736058 disk.dm-0.in‚10517.17578125 disk.sda3.in‚0" disk.nvme0n1.in‚10517.17578125 disk.sda1.out‚0 disk.drbd1260.out‚0.609375$ memory_percent‚98.93109798431396 proc.cpu.threads‚17iäÕ]¿‚(:„ ¾õêÔòφø gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚92.922' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.5844 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0àVPDqÀ‚(:æ ¾õêÔ­é¥ú disk.sda.in‚0" disk.nvme0n1.in‚10517.17578125 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.61328125 disk.dm-0.in‚10517.17578125 disk.sda.out‚ 128.53125 proc.memory.rssMB‚ 2924.59375* proc.memory.percent‚17.850303649902344 proc.cpu.threads‚17$ disk.nvme0c0n1.in‚10517.17578125 disk.sda1.in‚0 disk.dm-1.out‚ 128.53125 disk.sda3.in‚0 disk.sda3.out‚ 128.53125' proc.memory.availableMB‚ 174.8828125 network.recv‚ 1.476632e+06 disk.sda1.out‚0 disk.sda2.in‚0! disk.dm-0.out‚83108.263671875# memory_percent‚98.9326000213623 cpu‚25.37899113562837 disk.drbd1260.in‚0 network.sent‚ 6.375526e+06) disk./.usagePercent‚79.07277368475208$ disk./.usageGB‚1178.241470336914& disk.nvme0c0n1.out‚83108.263671875$ disk.nvme0n1.out‚83108.263671875 disk.sda2.out‚0èGr2Á- _wandbruntime‚4492  _runtime‚4492¾„ÆÆÂÀ _step‚210 & lr-AdamW/pg1‚0.0008823759469022628 & lr-AdamW/pg2‚0.0008823759469022628 trainer/global_step‚2599 $ _timestamp‚1.788525254343608e+09 _runtime‚4492.563380088,4MËÃÅ $ _timestamp‚1.788525254343608e+09 _runtime‚4492.563380088 _step‚210 & lr-AdamW/pg1‚0.0008823759469022628 & lr-AdamW/pg2‚0.0008823759469022628 trainer/global_step‚2599Òqì­2Ä- _wandbruntime‚4492  _runtime‚4492ÙÇò†ˆÅÖO WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5165884727808 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚112221026479616 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5151468900455 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5166118395904 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚680 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚117476373377536 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚49465621 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚105949704939520 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 120179570 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚70713947 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5255346897920 _runtime‚4492.564529013 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18787728 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚70713947 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29565852 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMn˜ºT onitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚112221002539520 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 120153225 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522752 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚111115589667328 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚105949722942464 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5151239919375 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚49465623 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚682 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7885252543444262e+09 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚111115841338368 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚105949704939520 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5255058375168 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5166118395904©‚ûwá'ÆÛO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚49465623 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚682 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7885252543444262e+09 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚111115841338368 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚105949704939520 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5255058375168 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5166118395904 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5165884727808 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚112221026479616 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5151468900455 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5166118395904 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚680 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚117476373377536 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚49465621 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚105949704939520 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 120179570 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚70713947 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5255346897920 _runtime‚4492.564529013 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18787728 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚70713947 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29565852 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚112221002539520 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 120153225 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522752 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚111115589667328 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚105949722942464 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5151239919375 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0Ó· õ2Ç- _wandbruntime‚4493  _runtime‚4493u3æ&ÈàM ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29576652 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚117519751290880 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚105988990976512 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚48371607 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48370938 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5257340707840 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 120224563 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚667 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 120198192 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚49483896 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚49483896 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚106004008884305 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚669 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚111157091414528 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5168082435072 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18794953 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 _runtime‚4493.682673554 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚49483898 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 120224563 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 120198192 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 % _timestamp‚1.7885252554621835e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚70714296 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚100850564455296 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚112262710449664 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚100850581494656 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5257064781312 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5153427389649 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚105988990976512 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚117520051157504 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 299866624 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚70740665 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚70740665 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚70714296 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5167848753664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 trainer/global_step‚2599 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚111157091414528 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5168082435072 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18794953 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 _runtime‚4493.682673554 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚49483898 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 120224563 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 120198192 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 % _timestamp‚1.7885252554621835e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚70714296 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚100850564455296 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚112262710449664 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚100850581494656 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5257064781312 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5153427389649 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚105988990976512 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚117520051157504 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 299866624 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚70740665 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚70740665 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚70714296 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5167848753664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 trainer/global_step‚2599 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚117519751290880 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚105988990976512 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚48371607 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48370938 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5257340707840 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 120224563 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚667 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 120198192 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚49483896 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚49483896 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚106004008884305 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚669 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚690 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚113178358894592 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚107938975949395 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30105952 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚114305224982016 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19132872 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚119660526077440 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5355012572672 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚102689569008432 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚72025556 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚107914024792064 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 _runtime‚4582.325718278 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚50379371 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚113178358894592 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚113178107223552 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5355301095424 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚107914006789120 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 122431272 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚107938729928955 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5249406940963 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5249177959883 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462848 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚690 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚113178358894592 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚107938975949395 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30105952 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚114305224982016 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19132872 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚119660526077440 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5355012572672 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚102689569008432 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚72025556 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚107914024792064 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 _runtime‚4582.325718278 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚50379371 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚113178358894592 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚113178107223552 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5355301095424 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚107914006789120 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 122431272 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚107938729928955 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5249406940963 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5249177959883 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462848 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7885253450627537e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚72052248 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚107978464796013 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 122449894 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5266298141696 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚72078617 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 308255232 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚119703895880704 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚114346885011968 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚709 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚711 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚113219608970752 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚50397648 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚50397646 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5251136448869 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 122476265 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30116752 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5357010868736 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚72052248 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚114346908952064 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚102727082326496 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚50397646 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5266298141696 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚107953292826112 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚107953292826112 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚107953310829056 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 122449894 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19140246 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 122476265 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚113219357286400 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚107953310829056 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49256289 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30116754 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5357295183872 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚107978218775365 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 _runtime‚4583.283168585 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚49257000 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49256289 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30116754 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5357295183872 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚107978218775365 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 _runtime‚4583.283168585 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚49257000 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7885253450627537e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚72052248 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚107978464796013 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 122449894 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5266298141696 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚72078617 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 308255232 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚119703895880704 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚114346885011968 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚709 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚711 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚113219608970752 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚50397648 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚50397646 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5251136448869 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 122476265 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30116752 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5357010868736 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚72052248 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚114346908952064 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚102727082326496 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚50397646 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5266298141696 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚107953292826112 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚107953292826112 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚107953310829056 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 122449894 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19140246 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 122476265 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚113219357286400 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚107953310829056 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273ظ–›P2â- _wandbruntime‚4583  _runtime‚4583°l@ªXãÒ .loss/FBt4/detection_loss‚0.765069488901645 trainer/global_step‚2649 0loss/FRt2/assignment_loss‚4.3045215959902166 /loss/FRt4/assignment_loss‚2.853274916963918 0loss/FBt3/assignment_loss‚0.6299309824179444 /loss/FRt4/detection_loss‚1.6189442970457353 .loss/FBt2/detection_loss‚2.223687776110389 /loss/FBt3/detection_loss‚2.0820221404234567 % _timestamp‚1.7885253450654907e+09 _runtime‚4583.284871304 0loss/FBt1/assignment_loss‚0.5206338187475207 /loss/FRt1/detection_loss‚2.7932218062250236 /loss/FRt2/detection_loss‚3.1488017726827553 /loss/FRt3/detection_loss‚2.2387843981937126 1loss/SRqqt2/assignment_loss‚2.720625827088952 1loss/FBt4/assignment_loss‚0.14324354205637652 1loss/SRqqt1/detection_loss‚1.9798278032491605 /loss/FRt1/assignment_loss‚3.836646970949675 /loss/FRt3/assignment_loss‚2.099673303189101 2loss/SRqqt1/assignment_loss‚1.2991473017415653 1loss/SRqqt4/detection_loss‚3.1099226474761963 2loss/SRqqt3/assignment_loss‚1.1332771808285744 /loss/FBt1/detection_loss‚2.4048018710953847 %loss/total_loss‚49.25316168007229 1loss/SRqqt3/detection_loss‚1.1950269818340462 _step‚217 1loss/SRqqt4/assignment_loss‚2.629946599988376 /loss/FBt2/assignment_loss‚0.203250075125156 1loss/SRqqt2/detection_loss‚3.3188785817474127 epoch‚2ÿͯÏ]ä× /loss/FBt1/detection_loss‚2.4048018710953847 %loss/total_loss‚49.25316168007229 1loss/SRqqt3/detection_loss‚1.1950269818340462 _step‚217 1loss/SRqqt4/assignment_loss‚2.629946599988376 /loss/FBt2/assignment_loss‚0.203250075125156 1loss/SRqqt2/detection_loss‚3.3188785817474127 epoch‚2 .loss/FBt4/detection_loss‚0.765069488901645 trainer/global_step‚2649 0loss/FRt2/assignment_loss‚4.3045215959902166 /loss/FRt4/assignment_loss‚2.853274916963918 0loss/FBt3/assignment_loss‚0.6299309824179444 /loss/FRt4/detection_loss‚1.6189442970457353 .loss/FBt2/detection_loss‚2.223687776110389 /loss/FBt3/detection_loss‚2.0820221404234567 % _timestamp‚1.7885253450654907e+09 _runtime‚4583.284871304 0loss/FBt1/assignment_loss‚0.5206338187475207 /loss/FRt1/detection_loss‚2.7932218062250236 /loss/FRt2/detection_loss‚3.1488017726827553 /loss/FRt3/detection_loss‚2.2387843981937126 1loss/SRqqt2/assignment_loss‚2.720625827088952 1loss/FBt4/assignment_loss‚0.14324354205637652 1loss/SRqqt1/detection_loss‚1.9798278032491605 /loss/FRt1/assignment_loss‚3.836646970949675 /loss/FRt3/assignment_loss‚2.099673303189101 2loss/SRqqt1/assignment_loss‚1.2991473017415653 1loss/SRqqt4/detection_loss‚3.1099226474761963 2loss/SRqqt3/assignment_loss‚1.1332771808285744Ù©(´rå‚(:ç §öêÔì›Øú network.recv‚ 1.50709e+06 disk.dm-1.out‚ 131.66796875 disk.drbd1260.in‚0 proc.memory.rssMB‚ 2924.59375% disk./.usageGB‚1177.2557754516602% disk.nvme0c0n1.out‚85300.75390625 disk.nvme0n1.in‚ 10773.859375# disk.nvme0n1.out‚85300.75390625 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.out‚ 0.62890625( proc.memory.availableMB‚ 176.38671875 network.sent‚ 6.517329e+06( disk./.usagePercent‚79.0066228739496 disk.sda2.in‚0 disk.dm-0.in‚ 10773.859375 disk.sda3.in‚0" disk.nvme0c0n1.in‚ 10773.859375 disk.dm-1.in‚0# memory_percent‚98.9234209060669 disk.sda2.out‚0 disk.dm-0.out‚85300.75390625 disk.sda.in‚0 disk.sda.out‚ 131.66796875 disk.sda3.out‚ 131.66796875* proc.memory.percent‚17.850303649902344 cpu‚25.376499798512636 proc.cpu.threads‚17¹¯ø‰‘æ‚(:† §öêÔ‘àú gpu.0.gpu‚15.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚103.316' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.6632 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ípç¡›ç‚(: ¶öêÔˆŽö gpu.0.gpu‚2.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚103.076' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚20.615199999999998 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0X¸¯sè‚(:è ¶öêÔ®¶Ö÷ disk.sda.out‚ 132.015625" disk.nvme0c0n1.in‚ 10773.859375' disk.nvme0c0n1.out‚85533.9443359375 disk.nvme0n1.in‚ 10773.859375 disk.dm-1.in‚0 network.recv‚ 1.512354e+06 disk.sda2.out‚0 disk.sda3.in‚0% disk.nvme0n1.out‚85533.9443359375 disk.drbd1260.out‚ 0.6328125$ memory_percent‚98.91889095306396( proc.memory.availableMB‚ 177.12890625 proc.memory.rssMB‚ 2924.59375 disk.sda2.in‚0 disk.sda3.out‚ 132.015625 disk.sda1.in‚0 disk.sda1.out‚0* proc.memory.percent‚17.850303649902344 cpu‚25.376233587649736% disk./.usageGB‚1177.2557754516602" disk.dm-0.out‚85533.9443359375 disk.sda.in‚0 disk.dm-1.out‚ 132.015625 disk.drbd1260.in‚0 proc.cpu.threads‚17 network.sent‚ 6.623954e+06( disk./.usagePercent‚79.0066228739496 disk.dm-0.in‚ 10773.859375?Ác‘é‚(:† ÅöêÔŸºÊö gpu.0.gpu‚17.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚102.541' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.5082 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÆB‡tê‚(:é ÅöêÔÁ ³ø network.sent‚ 6.629898e+06 network.recv‚ 1.516937e+06" disk.dm-0.out‚85806.3505859375 disk.sda3.in‚0" disk.nvme0c0n1.in‚ 10773.859375 disk.sda1.in‚0 disk.drbd1260.out‚ 0.6328125 disk.sda2.in‚0) disk./.usagePercent‚78.97748665647771% disk./.usageGB‚1176.8216247558594 disk.sda3.out‚ 132.421875' disk.nvme0c0n1.out‚85806.3505859375% disk.nvme0n1.out‚85806.3505859375 disk.sda2.out‚0( proc.memory.availableMB‚ 176.87890625 cpu‚25.375980458450318 disk.dm-0.in‚ 10773.859375 disk.sda.in‚0 disk.sda.out‚ 132.421875 disk.nvme0n1.in‚ 10773.859375 proc.memory.rssMB‚ 2924.59375 proc.cpu.threads‚17 disk.sda1.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 132.421875 disk.drbd1260.in‚0$ memory_percent‚98.92041683197021* proc.memory.percent‚17.850303649902344× ‘ë‚(:† ÔöêÔʼnˆû gpu.0.gpu‚18.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚101.601' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.3202 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÿOëfì‚(:Û ÔöêÔ힤ý cpu‚25.37568254804311 proc.cpu.threads‚17 network.recv‚ 1.522007e+06 disk.dm-1.in‚0 disk.sda2.out‚0 disk.sda.out‚ 132.84375 disk.sda3.out‚ 132.84375# proc.memory.availableMB‚176.875) disk./.usagePercent‚78.97748665647771 disk.nvme0n1.in‚ 10773.859375 disk.sda1.out‚0 disk.drbd1260.out‚ 0.63671875 disk.sda.in‚0 disk.sda3.in‚0$ memory_percent‚98.92044067382812 proc.memory.rssMB‚ 2924.59375 network.sent‚ 6.636237e+06% disk./.usageGB‚1176.8216247558594# disk.nvme0n1.out‚86440.30078125 disk.sda2.in‚0" disk.nvme0c0n1.in‚ 10773.859375* proc.memory.percent‚17.850303649902344 disk.sda1.in‚0 disk.dm-1.out‚ 132.84375 disk.drbd1260.in‚0 disk.dm-0.in‚ 10773.859375 disk.dm-0.out‚86440.30078125% disk.nvme0c0n1.out‚86440.30078125PWÒ‡‘í‚(:† ãöêÔµó„÷ gpu.0.gpu‚44.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚139.769' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.9538 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚03€„ iî‚(:Þ ãöêÔæ±â÷ disk.sda1.out‚0" memory_percent‚98.919677734375 proc.memory.availableMB‚177 disk.sda.in‚0* proc.memory.percent‚17.850303649902344 cpu‚25.375396269803968 proc.cpu.threads‚17 network.sent‚ 6.64113e+06" disk.dm-0.out‚86678.9892578125 disk.sda.out‚ 133.34375 disk.sda1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.63671875 disk.sda2.out‚0 proc.memory.rssMB‚ 2924.59375 disk.dm-0.in‚ 10931.2890625 disk.sda3.out‚ 133.34375# disk.nvme0c0n1.in‚ 10931.2890625' disk.nvme0c0n1.out‚86678.9892578125! disk.nvme0n1.in‚ 10931.2890625 disk.dm-1.in‚0 disk.dm-1.out‚ 133.34375 disk.sda2.in‚0 network.recv‚ 1.525624e+06) disk./.usagePercent‚78.97745311949072% disk./.usageGB‚1176.8211250305176 disk.sda3.in‚0% disk.nvme0n1.out‚86678.9892578125Açû‘ï‚(:† òöêÔö–ù gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚127.309' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.4618 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆPWyð‚(:î òöêÔÍÍ©ý* proc.memory.percent‚17.850303649902344 network.recv‚ 1.529317e+06) disk./.usagePercent‚78.97745311949072% disk./.usageGB‚1176.8211250305176 disk.sda3.out‚ 133.734375! disk.nvme0n1.in‚ 10931.2890625 disk.sda1.in‚0 disk.dm-1.out‚ 133.734375# disk.dm-0.out‚86840.51416015625# disk.nvme0c0n1.in‚ 10931.2890625 disk.sda.in‚0 cpu‚25.375041571907737 proc.cpu.threads‚17 disk.sda3.in‚0& disk.nvme0n1.out‚86840.51416015625 disk.sda1.out‚0 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚0.640625 disk.sda2.in‚0$ memory_percent‚98.91970157623291 network.sent‚ 6.646385e+06( disk.nvme0c0n1.out‚86840.51416015625 disk.sda2.out‚0 disk.dm-0.in‚ 10931.2890625 disk.sda.out‚ 133.734375( proc.memory.availableMB‚ 176.99609375 proc.memory.rssMB‚ 2924.59375éªÎV2ñ- _wandbruntime‚4671  _runtime‚4671èR?ÇòÁ _step‚218 & lr-AdamW/pg1‚0.0008770185459312784 & lr-AdamW/pg2‚0.0008770185459312784 trainer/global_step‚2699 % _timestamp‚1.7885254333675106e+09 _runtime‚4671.588117641ðÉ4ÍÌóÆ & lr-AdamW/pg1‚0.0008770185459312784 & lr-AdamW/pg2‚0.0008770185459312784 trainer/global_step‚2699 % _timestamp‚1.7885254333675106e+09 _runtime‚4671.588117641 _step‚218Úu3e‰2ô- _wandbruntime‚4671  _runtime‚46716‰‚õÖO TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚73363508 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚109913185840663 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚714 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚104566069840272 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 _step‚219 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚121844381105152 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5347116000391 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚115240624779776 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚51293123 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚51293121 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚116389423484416 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5362316141056 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚109878308638720 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5454981560832 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5362316141056 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚73363508 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚121844693568000 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚116389399544320 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚73389851 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚51293121 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚51293123 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5362549809152 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚50123401 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5455270083584 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30646052 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 _runtime‚4671.592410201 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5347344981471 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 trainer/global_step‚2699äÅŠEá'öÛO ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5362316141056 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚109878308638720 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5454981560832 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5362316141056 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚73363508 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚121844693568000 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚116389399544320 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚73389851 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚51293121 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚51293123 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5362549809152 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚50123401 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5455270083584 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30646052 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 _runtime‚4671.592410201 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5347344981471 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 trainer/global_step‚2699 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚73363508 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚109913185840663 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚714 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚104566069840272 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 _step‚219 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚121844381105152 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5347116000391 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚115240624779776 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚51293123 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚51293121 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚116389423484416 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773Û¡S<ç2÷- _wandbruntime‚4673  _runtime‚4673É”ü¬æ&øàM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19484009 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5364513848320 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50140187 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚109917612678656 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚115282126526976 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚676 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 _runtime‚4673.920744491 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5364280166912 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚674 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚109952674687073 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚51311396 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 trainer/global_step‚2699 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚73390200 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚51311396 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚73416569 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚109917612678656 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310352384 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚116431107454464 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 124727967 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286412288 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 124727967 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5349303470665 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚121888367587840 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 124701596 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚50140863 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5457260133376 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚115282126526976 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚51311398 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _step‚220 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚104603617237056 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚109917612678656 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚115282126526976 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚676 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 _runtime‚4673.920744491 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5364280166912 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚674 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚109952674687073 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚51311396 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 trainer/global_step‚2699 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚73390200 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚51311396 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚73416569 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚109917612678656 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310352384 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚116431107454464 BDeviceStatsMonitor.on_train_batch_end/allocation.all.ñ•™ðaallocated‚ 124727967 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286412288 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 124727967 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5349303470665 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚121888367587840 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 124701596 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚50140863 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5457260133376 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚115282126526976 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚51311398 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _step‚220 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚104603617237056 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50140187Ü#§p™2ú- _wandbruntime‚4673  _runtime‚4673žÉ]û× 1loss/FBt4/assignment_loss‚0.14781298146690947 trainer/global_step‚2699 .loss/FRt4/detection_loss‚1.963845580861044 &loss/total_loss‚52.814535407762875 /loss/FRt1/assignment_loss‚3.539568076531092 1loss/SRqqt2/detection_loss‚2.7336975468529596 1loss/SRqqt3/detection_loss‚0.9833702210912375 /loss/FBt3/detection_loss‚10.000721514225006 _runtime‚4673.926330278 2loss/SRqqt1/assignment_loss‚1.8279999860096723 2loss/SRqqt3/assignment_loss‚0.9260336754889023 2loss/SRqqt4/assignment_loss‚2.0065792846408756 /loss/FRt3/detection_loss‚1.5201555187148708 0loss/SRqqt1/detection_loss‚2.245981166139245 .loss/FRt4/assignment_loss‚2.88649213887178 3loss/FBt3/assignment_loss‚0.0007825411157682538 0loss/SRqqt4/detection_loss‚2.603410688313571 $ _timestamp‚1.788525435704241e+09 /loss/FRt1/detection_loss‚2.7906701034969754 0loss/FRt3/assignment_loss‚1.5027396730014255 2loss/SRqqt2/assignment_loss‚1.9432533995972738 1loss/FBt1/assignment_loss‚0.20711057592931562 1loss/FBt2/assignment_loss‚0.29523211053580667 /loss/FBt2/detection_loss‚2.3702798448503017 /loss/FBt4/detection_loss‚0.8416311319518303 0loss/FRt2/assignment_loss‚3.9464463194211326 -loss/FBt1/detection_loss‚2.67433739900589 epoch‚2 _step‚221 /loss/FRt2/detection_loss‚2.8563839296499887RL;büÜ /loss/FBt4/detection_loss‚0.8416311319518303 0loss/FRt2/assignment_loss‚3.9464463194211326 -loss/FBt1/detection_loss‚2.67433739900589 epoch‚2 _step‚221 /loss/FRt2/detection_loss‚2.8563839296499887 1loss/FBt4/assignment_loss‚0.14781298146690947 trainer/global_step‚2699 .loss/FRt4/detection_loss‚1.963845580861044 &loss/total_loss‚52.814535407762875 /loss/FRt1/assignment_loss‚3.539568076531092 1loss/SRqqt2/detection_loss‚2.7336975468529596 1loss/SRqqt3/detection_loss‚0.9833702210912375 /loss/FBt3/detection_loss‚10.000721514225006 _runtime‚4673.926330278 2loss/SRqqt1/assignment_loss‚1.8279999860096723 2loss/SRqqt3/assignment_loss‚0.9260336754889023 2loss/SRqqt4/assignment_loss‚2.0065792846408756 /loss/FRt3/detection_loss‚1.5201555187148708 0loss/SRqqt1/detection_loss‚2.245981166139245 .loss/FRt4/assignment_loss‚2.88649213887178 3loss/FBt3/assignment_loss‚0.0007825411157682538 0loss/SRqqt4/detection_loss‚2.603410688313571 $ _timestamp‚1.788525435704241e+09 /loss/FRt1/detection_loss‚2.7906701034969754 0loss/FRt3/assignment_loss‚1.5027396730014255 2loss/SRqqt2/assignment_loss‚1.9432533995972738 1loss/FBt1/assignment_loss‚0.20711057592931562 1loss/FBt2/assignment_loss‚0.29523211053580667 /loss/FBt2/detection_loss‚2.3702798448503017Ýz C‘ý‚(:† ÷êÔ¢òÇ÷ gpu.0.gpu‚32.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚111.526' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.3052 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¶ìäxþ‚(:í ÷êÔÖåù disk.dm-0.in‚ 10931.2890625 disk.sda.in‚0 disk.sda.out‚ 134.078125# memory_percent‚98.9201307296753( proc.memory.availableMB‚ 176.92578125 disk.sda1.out‚0 disk.sda3.out‚ 134.078125 proc.memory.rssMB‚ 2924.59375* proc.memory.percent‚17.850303649902344 proc.cpu.threads‚17 network.sent‚ 6.652155e+06 network.recv‚ 1.533851e+06% disk./.usageGB‚1176.8215827941895& disk.nvme0n1.out‚87071.13916015625 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚0) disk./.usagePercent‚78.97748384039483# disk.nvme0c0n1.in‚ 10931.2890625( disk.nvme0c0n1.out‚87071.13916015625 disk.sda1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚0.640625# disk.dm-0.out‚87071.13916015625 cpu‚25.374744988607954! disk.nvme0n1.in‚ 10931.2890625 disk.dm-1.in‚0 disk.dm-1.out‚ 134.078125»G]‰‘ÿ‚(:† ÷êÔªª¾÷ gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚33.0 gpu.0.powerWatts‚139.119' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.8238 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0뾪ø~€‚(:ó ÷êÔ³ƒ‰ú disk.dm-1.out‚ 134.44140625 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda.out‚ 134.44140625 disk.sda3.in‚0( proc.memory.availableMB‚ 176.92578125* proc.memory.percent‚17.850303649902344 cpu‚25.37381319656008 network.sent‚ 6.762691e+06 network.recv‚ 1.540444e+06% disk./.usageGB‚1176.1819229125977& disk.nvme0n1.out‚87630.41064453125 disk.sda1.in‚0 disk.dm-1.in‚0 disk.dm-0.in‚ 10931.2890625# disk.dm-0.out‚87630.41064453125( disk./.usagePercent‚78.9345557290308# disk.nvme0c0n1.in‚ 10931.2890625( disk.nvme0c0n1.out‚87630.41064453125 disk.sda1.out‚0 disk.drbd1260.out‚ 0.64453125 disk.sda3.out‚ 134.44140625 proc.memory.rssMB‚ 2924.59375 proc.cpu.threads‚17! disk.nvme0n1.in‚ 10931.2890625 disk.drbd1260.in‚0 disk.sda.in‚0# memory_percent‚98.9201307296753¥Ë‚›‚(: Ÿ÷êÔèÞ»ö gpu.0.gpu‚35.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚112.34' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.468000000000004 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0f7·B‚‚‚(:÷ Ÿ÷êÔ¢Åù÷ disk.sda3.in‚0( disk.nvme0c0n1.out‚87971.21533203125 disk.sda1.out‚0$ memory_percent‚98.92017841339111 proc.memory.rssMB‚ 2924.59375 proc.cpu.threads‚17) disk./.usagePercent‚78.93534423223628 disk.sda2.out‚0# disk.dm-0.out‚87971.21533203125& disk.nvme0n1.out‚87971.21533203125 disk.dm-1.in‚0 disk.dm-1.out‚ 134.91015625 disk.drbd1260.out‚ 0.64453125( proc.memory.availableMB‚ 176.91796875 network.recv‚ 1.543745e+06% disk./.usageGB‚1176.1936721801758 disk.sda.in‚0 disk.sda3.out‚ 134.91015625 disk.drbd1260.in‚0 cpu‚25.3729213918224 network.sent‚ 6.765745e+06 disk.dm-0.in‚11075.63671875$ disk.nvme0c0n1.in‚11075.63671875" disk.nvme0n1.in‚11075.63671875 disk.sda1.in‚0* proc.memory.percent‚17.850303649902344 disk.sda2.in‚0 disk.sda.out‚ 134.91015625 i𑃂(:† ®÷êÔ¦ô…÷ gpu.0.gpu‚30.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚118.114' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.6228 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0UÜ|„‚(:ñ ®÷êÔû ø proc.memory.rssMB‚ 2924.59375 proc.cpu.threads‚17' disk.nvme0c0n1.out‚88230.6923828125 network.recv‚ 1.547354e+06% disk./.usageGB‚1175.9524765014648 disk.sda.out‚ 135.26953125 disk.sda3.in‚0" disk.nvme0n1.in‚11075.63671875 disk.drbd1260.in‚0# memory_percent‚98.9201545715332$ disk.nvme0c0n1.in‚11075.63671875 disk.drbd1260.out‚ 0.6484375& proc.memory.availableMB‚ 176.921875* proc.memory.percent‚17.850303649902344 network.sent‚ 6.77091e+06) disk./.usagePercent‚78.91915738786133 disk.sda2.out‚0" disk.dm-0.out‚88230.6923828125% disk.nvme0n1.out‚88230.6923828125 disk.sda1.in‚0 disk.dm-1.in‚0 cpu‚25.372113238236835 disk.sda2.in‚0 disk.dm-0.in‚11075.63671875 disk.sda.in‚0 disk.sda3.out‚ 135.26953125 disk.sda1.out‚0 disk.dm-1.out‚ 135.26953125$ÍkR2…- _wandbruntime‚4726  _runtime‚4726öiu!džÁ trainer/global_step‚2749 % _timestamp‚1.7885254881787553e+09 _runtime‚4726.398265511 _step‚222 & lr-AdamW/pg1‚0.0008742090278214344 & lr-AdamW/pg2‚0.0008742090278214344åVO0Ì‡Æ _step‚222 & lr-AdamW/pg1‚0.0008742090278214344 & lr-AdamW/pg2‚0.0008742090278214344 trainer/global_step‚2749 % _timestamp‚1.7885254881787553e+09 _runtime‚4726.398265511Þ¦9',2ˆ- _wandbruntime‚4726  _runtime‚4726Zû•úÛ'‰ÕO XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5460765515776 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚117303142336000 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚111842628491264 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚51010726 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚118473621986816 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5460531847680 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚685 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462848 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚52206871 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31186152 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5445054040899 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚111842610488320 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚52206871 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚74701460 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚117303394007040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚124028519864320 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚111887887772811 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5554921817600 $ _timestamp‚1.788525488179818e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚117303394007040 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 126934676 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5445283021979 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚111842628491264 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚74727803 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚687 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5460765515776 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19825261 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522752 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚74701460 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5460531847680 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚106442587711472 trainer/global_step‚2749 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19824576 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚117303142336000 _runtime‚4726.399562526 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31186150 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚52206873t¯å‚!ŠÚO H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5460765515776 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚117303142336000 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚111842628491264 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚51010726 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚118473621986816 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5460531847680 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚685 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462848 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚52206871 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31186152 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5445054040899 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚111842610488320 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚52206871 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚74701460 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚117303394007040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚124028519864320 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚111887887772811 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5554921817600 $ _timestamp‚1.788525488179818e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚117303394007040 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 126934676 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5445283021979 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚111842628491264 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚74727803 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚687 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5460765515776 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19825261 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522752 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚74701460 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5460531847680 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚106442587711472 trainer/global_step‚2749 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19824576 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚117303142336000 _runtime‚4726.399562526 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31186150 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚52206873 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608ßEÞt2‹- _wandbruntime‚4727  _runtime‚4727–Wâ5å&ŒßM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚52225148 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5557224105984 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5462729554944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31196952 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19831831 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5447012529885 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 126953298 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚117344392398848 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚2749 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚111881896525312 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚52225148 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51028783 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚111927376619429 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5462495873536 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290606592 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚684 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5462495873536 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚111881914528256 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚682 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5556933499392 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚124072215516160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚74754521 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19832513 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚118515282016768 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 126953298 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚74728152 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚106480135108256 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚118515305956864 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚52225146 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚124072530062848 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚111881896525312 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚117344644083200 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚74754521 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5462729554944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚74728152 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 _step‚224 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546688 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 126979669 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 126979669 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚106480118068896 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚52225146 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚111927130598781 _runtime‚ 4727.35767547 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚111881914528256 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚117344644083200 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0¢éÝ¢ê&äM SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚111927130598781 _runtime‚ 4727.35767547 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚111881914528256 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚117344644083200 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚52225148 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5557224105984 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5462729554944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31196952 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19831831 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5447012529885 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 126953298 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚117344392398848 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚2749 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚111881896525312 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚52225148 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51028783 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚111927376619429 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5462495873536 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 290606592 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚684 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5462495873536 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚111881914528256 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚682 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5556933499392 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚124072215516160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚74754521 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19832513 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚118515282016768 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 126953298 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚74728152 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚106480135108256 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚118515305956864 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚52225146 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚124072530062848 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚111881896525312 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚117344644083200 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚74754521 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5462729554944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚74728152 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 _step‚224 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546688 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 126979669 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 126979669 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚106480118068896 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚52225146 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288àžÜu«2Ž- _wandbruntime‚4727  _runtime‚4727¹Nê®YÓ .loss/FBt3/detection_loss‚3.044847011566162 $loss/total_loss‚47.5036401104513 /loss/FRt1/assignment_loss‚4.024037842092843 2loss/SRqqt3/assignment_loss‚1.1339705166337657 1loss/FBt2/assignment_loss‚0.36369380077998026 0loss/SRqqt1/detection_loss‚2.250359834117048 0loss/FBt4/assignment_loss‚0.2050110662398304 /loss/FBt2/detection_loss‚2.1899647306312215 1loss/SRqqt3/detection_loss‚0.9545007475520954 .loss/FBt1/detection_loss‚2.330602240562439 /loss/FBt4/detection_loss‚0.7132561241961557 _runtime‚4727.358510113 /loss/FRt4/assignment_loss‚3.353372508186405 2loss/SRqqt1/assignment_loss‚1.9736085345639902 epoch‚2 2loss/SRqqt2/assignment_loss‚2.0095193314437685 0loss/FBt1/assignment_loss‚0.4266352377730186 /loss/FRt3/detection_loss‚1.9810297050409846 1loss/SRqqt2/detection_loss‚2.4966080647248488 0loss/FRt2/assignment_loss‚3.7057018961225237 /loss/FRt3/assignment_loss‚1.721080517327344 2loss/SRqqt4/assignment_loss‚2.1181157932561985 .loss/FRt2/detection_loss‚3.246652779125032 1loss/SRqqt4/detection_loss‚2.7707847181488487 _step‚225 1loss/FBt3/assignment_loss‚0.12116257012821734 /loss/FRt1/detection_loss‚2.7161150924090682 /loss/FRt4/detection_loss‚1.6530094478295136 trainer/global_step‚2749 % _timestamp‚1.7885254891391518e+09^Ø 1loss/SRqqt2/detection_loss‚2.4966080647248488 0loss/FRt2/assignment_loss‚3.7057018961225237 /loss/FRt3/assignment_loss‚1.721080517327344 2loss/SRqqt4/assignment_loss‚2.1181157932561985 .loss/FRt2/detection_loss‚3.246652779125032 1loss/SRqqt4/detection_loss‚2.7707847181488487 _step‚225 1loss/FBt3/assignment_loss‚0.12116257012821734 /loss/FRt1/detection_loss‚2.7161150924090682 /loss/FRt4/detection_loss‚1.6530094478295136 trainer/global_step‚2749 % _timestamp‚1.7885254891391518e+09 .loss/FBt3/detection_loss‚3.044847011566162 $loss/total_loss‚47.5036401104513 /loss/FRt1/assignment_loss‚4.024037842092843 2loss/SRqqt3/assignment_loss‚1.1339705166337657 1loss/FBt2/assignment_loss‚0.36369380077998026 0loss/SRqqt1/detection_loss‚2.250359834117048 0loss/FBt4/assignment_loss‚0.2050110662398304 /loss/FBt2/detection_loss‚2.1899647306312215 1loss/SRqqt3/detection_loss‚0.9545007475520954 .loss/FBt1/detection_loss‚2.330602240562439 /loss/FBt4/detection_loss‚0.7132561241961557 _runtime‚4727.358510113 /loss/FRt4/assignment_loss‚3.353372508186405 2loss/SRqqt1/assignment_loss‚1.9736085345639902 epoch‚2 2loss/SRqqt2/assignment_loss‚2.0095193314437685 0loss/FBt1/assignment_loss‚0.4266352377730186 /loss/FRt3/detection_loss‚1.9810297050409846á;X‡‘‚(:‚ ½÷êÔ»ûö gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.14' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.628 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»Œƒì~’‚(:ó ½÷êÔЖì÷ disk.sda1.in‚0 disk.dm-1.out‚ 135.59765625 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.6484375 disk.sda2.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0 disk.sda2.out‚0 disk.dm-0.in‚11075.63671875' proc.memory.availableMB‚ 176.8828125 network.sent‚ 6.776824e+06 network.recv‚ 1.551855e+06% disk./.usageGB‚1175.2169799804688" disk.nvme0n1.in‚11075.63671875 proc.memory.rssMB‚ 2924.59375* proc.memory.percent‚17.850303649902344 cpu‚25.371526717990907) disk./.usagePercent‚78.86979759921459 disk.sda.out‚ 135.59765625 disk.sda3.in‚0 disk.sda3.out‚ 135.59765625$ disk.nvme0c0n1.in‚11075.63671875" disk.dm-0.out‚88431.5283203125# memory_percent‚98.9203929901123 proc.cpu.threads‚17 disk.sda.in‚0' disk.nvme0c0n1.out‚88431.5283203125% disk.nvme0n1.out‚88431.5283203125å ÄJ“‚(:… Ì÷êÔ™ ðö gpu.0.gpu‚21.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚114.725' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.945 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0!ÝB‚”‚(:÷ Ì÷êÔ݉º÷ disk.sda3.in‚0# memory_percent‚98.9203691482544 network.recv‚ 1.558307e+06) disk./.usagePercent‚78.86308021752366$ disk./.usageGB‚1175.116886138916# disk.dm-0.out‚88581.46533203125 disk.sda.out‚ 136.01953125 disk.sda3.out‚ 136.01953125 proc.memory.rssMB‚ 2924.59375* proc.memory.percent‚17.850303649902344( disk.nvme0c0n1.out‚88581.46533203125 disk.sda2.in‚0 disk.dm-0.in‚11075.63671875 cpu‚25.371151784272687 proc.cpu.threads‚17 network.sent‚ 6.885509e+06 disk.sda1.in‚0 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda2.out‚0 disk.sda.in‚0( proc.memory.availableMB‚ 176.88671875$ disk.nvme0c0n1.in‚11075.63671875" disk.nvme0n1.in‚11075.63671875& disk.nvme0n1.out‚88581.46533203125 disk.sda1.out‚0 disk.dm-1.out‚ 136.01953125 disk.drbd1260.out‚ 0.65234375Í'ÑÈœ•‚(:‘ Û÷êÔ‡ã˜ö gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚104.543' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚20.908600000000003 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Bãg~–‚(:ó Û÷êÔ×æø network.sent‚ 6.890472e+06( disk.nvme0c0n1.out‚88806.62939453125& disk.nvme0n1.out‚88806.62939453125 disk.dm-0.in‚ 11207.8046875# disk.dm-0.out‚88806.62939453125 disk.sda.out‚ 136.57421875 disk.sda3.in‚0 disk.sda3.out‚ 136.57421875! disk.nvme0n1.in‚ 11207.8046875 disk.sda1.out‚0 disk.dm-1.out‚ 136.57421875 disk.drbd1260.in‚0 disk.sda2.out‚0' proc.memory.availableMB‚ 176.9453125 proc.memory.rssMB‚ 2924.59375* proc.memory.percent‚17.850303649902344 network.recv‚ 1.561545e+06( disk./.usagePercent‚78.8683657490756# disk.nvme0c0n1.in‚ 11207.8046875 disk.sda1.in‚0$ memory_percent‚98.92001152038574 cpu‚25.370301923379774 proc.cpu.threads‚17$ disk./.usageGB‚1175.195644378662 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.65234375 disk.sda2.in‚0 disk.sda.in‚0y8 c‘—‚(:† ê÷êÔú¸¨õ gpu.0.gpu‚27.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚106.711' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.3422 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0\‰ÐÒy˜‚(:î ê÷êÔ•Øö÷ disk.sda.in‚0 disk.sda3.out‚ 136.9296875# disk.nvme0c0n1.in‚ 11207.8046875! disk.nvme0n1.in‚ 11207.8046875 proc.memory.rssMB‚ 2924.59375 cpu‚25.369671307665175 proc.cpu.threads‚17 network.sent‚ 6.897503e+06) disk./.usagePercent‚78.87207299417892 disk.drbd1260.in‚0 disk.sda3.in‚0$ memory_percent‚98.92148971557617* proc.memory.percent‚17.850303649902344 network.recv‚ 1.565175e+06 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda.out‚ 136.9296875( disk.nvme0c0n1.out‚89372.66357421875& disk.nvme0n1.out‚89372.66357421875& proc.memory.availableMB‚ 176.703125% disk./.usageGB‚1175.2508850097656 disk.sda1.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 136.9296875 disk.drbd1260.out‚0.65625 disk.sda2.in‚0 disk.dm-0.in‚ 11207.8046875# disk.dm-0.out‚89372.66357421875–˜Ø†2™- _wandbruntime‚4796  _runtime‚4796ÿdB+ÇšÁ trainer/global_step‚2799 % _timestamp‚1.7885255586454928e+09 _runtime‚4796.865020631 _step‚226 & lr-AdamW/pg1‚0.0008713130725827116 & lr-AdamW/pg2‚0.0008713130725827116þÂ3 Ì›Æ _step‚226 & lr-AdamW/pg1‚0.0008713130725827116 & lr-AdamW/pg2‚0.0008713130725827116 trainer/global_step‚2799 % _timestamp‚1.7885255586454928e+09 _runtime‚4796.865020631â sY2œ- _wandbruntime‚4796  _runtime‚4796æDhÝŽ ÖO ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚126212572900352 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5654776351232 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5558981222400 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚53120623 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚108319122622032 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5558747554304 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚701 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 _runtime‚4796.866208844 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚53120623 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5655066971136 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚76065755 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚113806930340864 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚119365659892224 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5558981222400 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚113862343684519 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚108319105582672 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 129186378 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚699 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚113862097664079 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚113806912337920 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚53120621 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 314560000 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51902242 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5542992081407 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚126212887460352 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 _step‚227 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚113806912337920 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5558747554304 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚120557796549120 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚76039412 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20175990 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 % _timestamp‚1.7885255586465626e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚119365911563264 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚119365911563264 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚76039412 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5543221062487 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 129160033 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚76065755 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5655066971136 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚76065755 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚113806930340864 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚119365659892224 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5558981222400 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚113862343684519 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚108319105582672 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 129186378 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚699 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚113862097664079 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚113806912337920 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚53120621 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 314560000 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944ã®}Š2Ÿ- _wandbruntime‚4797  _runtime‚4797}¡¿æ& àM MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚76066104 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚120599480519168 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5657080015872 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5656789409280 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5545179551681 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚76092473 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 129231371 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 129205000 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚677 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚113901832531137 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚126256584475136 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 129231371 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚113846198374912 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚51919719 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51919040 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31737052 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚126256269928448 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚113846216377856 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚76066104 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚113846216377856 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚53138896 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚120599504459264 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚113901586510489 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚119406909955072 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚119407161639424 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546688 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7885255596173253e+097¹®–¡åM ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚113846198374912 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚51919719 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51919040 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31737052 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚126256269928448 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚113846216377856 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚76066104 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚113846216377856 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚53138896 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚120599504459264 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚113901586510489 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚119406909955072 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚119407161639424 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546688 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7885255596173253e+09 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚76066104 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚120599480519168 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5657080015872 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5656789409280 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5545179551681 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚76092473 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 129231371 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 129205000 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚677 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚113901832531137 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚126256584475136 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 129231371 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280äêëw 2¢- _wandbruntime‚4797  _runtime‚4797‘#ªR£Ì /loss/FRt3/detection_loss‚1.6833531161504132 trainer/global_step‚2799 /loss/FRt2/detection_loss‚2.9163787722587586 0loss/FRt1/assignment_loss‚3.2172152185440064 ,loss/FRt3/assignment_loss‚1.617537854399 1loss/SRqqt2/assignment_loss‚2.074208809206119 1loss/FBt2/assignment_loss‚0.39923325674183413 1loss/FBt4/assignment_loss‚0.24727981055489534 .loss/FBt1/detection_loss‚2.231251547336578 1loss/SRqqt1/assignment_loss‚1.429243329167366 .loss/FBt1/assignment_loss‚0.42061507266435 /loss/FRt4/detection_loss‚1.8906153009738773 1loss/SRqqt3/detection_loss‚0.9923745792681776 %loss/total_loss‚45.89735246023025 2loss/SRqqt3/assignment_loss‚1.1211584228142275 1loss/SRqqt1/detection_loss‚2.6401491069793703 epoch‚2 _runtime‚4797.838747336 0loss/FRt4/assignment_loss‚2.7129509099598588 /loss/FBt3/detection_loss‚3.3881172140439353 .loss/FBt2/detection_loss‚2.327622413635254 /loss/FBt4/detection_loss‚1.0179391323365796 $ _timestamp‚1.788525559619454e+09 0loss/SRqqt4/detection_loss‚2.459992993209097 2loss/SRqqt4/assignment_loss‚1.6578499428513978 /loss/FBt3/assignment_loss‚0.008235846660682 _step‚229 /loss/FRt2/assignment_loss‚3.872335320711136 /loss/FRt1/detection_loss‚3.0374495363235474 0loss/SRqqt2/detection_loss‚2.534244953439786Ûˆ0¥W¤Ñ 2loss/SRqqt4/assignment_loss‚1.6578499428513978 /loss/FBt3/assignment_loss‚0.008235846660682 _step‚229 /loss/FRt2/assignment_loss‚3.872335320711136 /loss/FRt1/detection_loss‚3.0374495363235474 0loss/SRqqt2/detection_loss‚2.534244953439786 /loss/FRt3/detection_loss‚1.6833531161504132 trainer/global_step‚2799 /loss/FRt2/detection_loss‚2.9163787722587586 0loss/FRt1/assignment_loss‚3.2172152185440064 ,loss/FRt3/assignment_loss‚1.617537854399 1loss/SRqqt2/assignment_loss‚2.074208809206119 1loss/FBt2/assignment_loss‚0.39923325674183413 1loss/FBt4/assignment_loss‚0.24727981055489534 .loss/FBt1/detection_loss‚2.231251547336578 1loss/SRqqt1/assignment_loss‚1.429243329167366 .loss/FBt1/assignment_loss‚0.42061507266435 /loss/FRt4/detection_loss‚1.8906153009738773 1loss/SRqqt3/detection_loss‚0.9923745792681776 %loss/total_loss‚45.89735246023025 2loss/SRqqt3/assignment_loss‚1.1211584228142275 1loss/SRqqt1/detection_loss‚2.6401491069793703 epoch‚2 _runtime‚4797.838747336 0loss/FRt4/assignment_loss‚2.7129509099598588 /loss/FBt3/detection_loss‚3.3881172140439353 .loss/FBt2/detection_loss‚2.327622413635254 /loss/FBt4/detection_loss‚1.0179391323365796 $ _timestamp‚1.788525559619454e+09 0loss/SRqqt4/detection_loss‚2.459992993209097å»6KÑ¥‚(:† ù÷êÔž£Ì÷ gpu.0.gpu‚30.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚109.246' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.8492 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Õ´Nøf¦‚(:Û ù÷êÔ»â·ù! disk.nvme0n1.in‚ 11207.8046875$ memory_percent‚98.92797470092773& proc.memory.availableMB‚ 175.640625% disk./.usageGB‚1175.2743530273438 disk.dm-1.out‚137.5 disk.sda3.in‚0& disk.nvme0n1.out‚89565.15966796875 proc.memory.rssMB‚ 2924.59375 proc.cpu.threads‚17 network.recv‚ 1.569456e+06 disk.sda2.in‚0 disk.dm-0.in‚ 11207.8046875 disk.sda.out‚137.5* proc.memory.percent‚17.850303649902344 cpu‚25.36938787333695 network.sent‚ 6.901547e+06 disk.sda1.in‚0 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda2.out‚0 disk.sda.in‚0 disk.sda3.out‚137.5# disk.nvme0c0n1.in‚ 11207.8046875) disk./.usagePercent‚78.87364795252957 disk.sda1.out‚0 disk.drbd1260.out‚0.65625# disk.dm-0.out‚89565.15966796875( disk.nvme0c0n1.out‚89565.15966796875Ó¹(&w§‚(:ì ˆøêÔÏ„Àù$ memory_percent‚98.92051219940186( proc.memory.availableMB‚ 176.86328125 disk.dm-1.in‚0 disk.sda3.out‚ 138.0234375& disk.nvme0c0n1.out‚89570.705078125! disk.nvme0n1.in‚ 11207.8046875 disk.sda1.in‚0 proc.memory.rssMB‚ 2924.59375$ disk./.usageGB‚1175.279167175293$ disk.nvme0n1.out‚89570.705078125 disk.sda1.out‚0* proc.memory.percent‚17.850303649902344 cpu‚25.368813597993626 proc.cpu.threads‚17 network.sent‚ 7.010269e+06 network.recv‚ 1.576233e+06) disk./.usagePercent‚78.87397103403778 disk.dm-1.out‚ 138.0234375 disk.drbd1260.out‚ 0.66015625 disk.dm-0.in‚ 11207.8046875! disk.dm-0.out‚89570.705078125 disk.sda3.in‚0 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda.in‚0 disk.sda.out‚ 138.0234375# disk.nvme0c0n1.in‚ 11207.8046875Æ¥;‘¨‚(:† ˆøêÔ§íÏù gpu.0.gpu‚22.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚126.783' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.3566 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0äèä-z©‚(:ï —øêÔ›õø÷& disk.nvme0c0n1.out‚89777.279296875 disk.dm-1.in‚0 proc.memory.rssMB‚ 2924.59375* proc.memory.percent‚17.850303649902344 cpu‚25.3685167131897 network.sent‚ 7.01532e+06% disk./.usageGB‚1175.3415870666504 disk.dm-0.in‚11363.39453125 disk.sda3.in‚0 disk.sda3.out‚ 138.40234375 disk.dm-1.out‚ 138.40234375 disk.drbd1260.out‚ 0.66015625 disk.sda2.in‚0 disk.sda.in‚0$ disk.nvme0c0n1.in‚11363.39453125" disk.nvme0n1.in‚11363.39453125 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0# memory_percent‚98.9220380783081( proc.memory.availableMB‚ 176.61328125 network.recv‚ 1.579624e+06) disk./.usagePercent‚78.87816008532054! disk.dm-0.out‚89777.279296875 disk.sda.out‚ 138.40234375$ disk.nvme0n1.out‚89777.279296875 disk.drbd1260.in‚0 proc.cpu.threads‚17Õi@ôª‚(:… —øêÔ£˜ðù gpu.0.gpu‚22.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚102.715' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.543 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0†ùê•‘«‚(:† ¦øêÔ—ý½ö gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚138.346' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.6692 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡0ôt¬‚(:é ¦øêԼ̥÷ cpu‚25.36805849030054 network.sent‚ 7.020412e+06% disk./.usageGB‚1175.4040870666504 disk.sda3.out‚ 138.92578125$ disk.nvme0c0n1.out‚ 89932.0078125" disk.nvme0n1.in‚11363.39453125 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.sda.in‚0 network.recv‚ 1.583365e+06$ disk.nvme0c0n1.in‚11363.39453125" disk.nvme0n1.out‚ 89932.0078125 disk.dm-1.in‚0 disk.sda.out‚ 138.92578125' proc.memory.availableMB‚ 176.6171875* proc.memory.percent‚17.850303649902344) disk./.usagePercent‚78.88235451276154 disk.sda1.in‚0 disk.dm-1.out‚ 138.92578125 disk.sda2.out‚0 disk.dm-0.in‚11363.39453125 disk.dm-0.out‚ 89932.0078125 proc.cpu.threads‚17 disk.sda3.in‚0 disk.sda1.out‚0 disk.drbd1260.out‚ 0.6640625# memory_percent‚98.9220142364502 proc.memory.rssMB‚ 2924.59375 £x4­‚(:„ µøêÔ™²Ì÷ gpu.0.gpu‚4.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚108.585' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.717 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0úUér®‚(:ç µøêÔàˆÈù network.recv‚ 1.587635e+06 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.out‚90182.45703125$ memory_percent‚98.92206192016602& proc.memory.availableMB‚ 176.609375 cpu‚25.367775402012555 proc.cpu.threads‚17 network.sent‚ 7.026262e+06$ disk.nvme0c0n1.in‚11363.39453125" disk.nvme0n1.in‚11363.39453125# disk.nvme0n1.out‚90182.45703125 disk.sda1.out‚0 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda.out‚ 139.265625) disk./.usagePercent‚78.88916584921755% disk.nvme0c0n1.out‚90182.45703125 disk.sda1.in‚0 disk.dm-1.out‚ 139.265625 disk.drbd1260.out‚ 0.6640625 disk.dm-0.in‚11363.39453125 disk.sda3.in‚0 disk.sda3.out‚ 139.265625% disk./.usageGB‚1175.5055809020996 disk.sda.in‚0 proc.memory.rssMB‚ 2924.59375* proc.memory.percent‚17.850303649902344ì[÷‘¯‚(:† ÄøêԙΊù gpu.0.gpu‚27.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚111.232' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.2464 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0e5©”ƒ°‚(:ø ÄøêÔ£¯Çú network.recv‚ 1.592539e+06$ disk.nvme0c0n1.in‚11363.39453125& disk.nvme0n1.out‚90581.48486328125 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.dm-0.in‚11363.39453125 disk.sda3.out‚ 139.65234375( proc.memory.availableMB‚ 176.60546875" disk.nvme0n1.in‚11363.39453125 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.66796875# disk.dm-0.out‚90581.48486328125 disk.sda.out‚ 139.65234375* proc.memory.percent‚17.850303649902344 cpu‚25.367406208006546 proc.cpu.threads‚17% disk./.usageGB‚1175.7566375732422( disk.nvme0c0n1.out‚90581.48486328125 disk.sda1.in‚0 disk.sda2.out‚0$ memory_percent‚98.92208576202393 proc.memory.rssMB‚ 2924.59375 network.sent‚ 7.032592e+06( disk./.usagePercent‚78.9060144730685 disk.dm-1.out‚ 139.65234375 disk.sda2.in‚0 disk.sda.in‚0 disk.sda3.in‚0p5»W2±- _wandbruntime‚4876  _runtime‚4876ßïDzÁ _runtime‚4876.995234636 _step‚230 & lr-AdamW/pg1‚0.0008683312837278411 & lr-AdamW/pg2‚0.0008683312837278411 trainer/global_step‚2849 % _timestamp‚1.7885256387750218e+09x\N]Ì³Æ % _timestamp‚1.7885256387750218e+09 _runtime‚4876.995234636 _step‚230 & lr-AdamW/pg1‚0.0008683312837278411 & lr-AdamW/pg2‚0.0008683312837278411 trainer/global_step‚2849æ<§pš2´- _wandbruntime‚4877  _runtime‚4877úAúÜ'µÖO `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚115771232190464 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5656963260928 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚122642018991616 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5754597090816 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚110195623453872 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚115771214187520 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚121428429119488 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚115771214187520 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚704 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20522156 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 310365696 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚702 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚77377364 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32266350 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32266352 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 131411735 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚54034373 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚128396902508032 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 trainer/global_step‚2849 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚121428177448448 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚52787804 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚5823‡5@¶ÛO E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚54034371 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5657196929024 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚54034373 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5656963260928 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5754883516416 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚54034371 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚122642018991616 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5754597090816 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚110195623453872 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚115771214187520 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚121428429119488 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚115771214187520 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚704 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20522156 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 310365696 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚702 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚77377364 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32266350 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32266352 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 131411735 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚54034373 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚128396902508032 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 trainer/global_step‚2849 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚121428177448448 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚52787804 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚115771232190464 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5656963260928 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5659160968192 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚52805871 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5659160968192 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚110233170850656 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5756881731584 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚77404056 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52805162 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚115876042422197 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 131483073 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚54052648 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚115810518227456 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 131483073 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20528717 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚121469679195648 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚115810500224512 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚110233153811296 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 131456702 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚115810518227456 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 296898048 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 % _timestamp‚1.7885256412244103e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚122683679021568 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚121469679195648 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 131456702 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32277154 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚54052646 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32277152 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚121469427511296 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚54052648 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 320838144 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273h»M©ë&¹åM WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚121469679195648 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 131456702 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32277154 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚54052646 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32277152 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚121469427511296 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚54052648 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 320838144 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5658927286784 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5642888610901 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚54052646 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚77430425 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5756584833536 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 trainer/global_step‚2849 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚77404056 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚128440584693248 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚122683702961664 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5659160968192 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚52805871 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5659160968192 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚110233170850656 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5756881731584 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚77404056 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52805162 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚115876042422197 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 131483073 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚54052648 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚115810518227456 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 131483073 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20528717 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚121469679195648 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚115810500224512 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚110233153811296 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 131456702 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚115810518227456 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 296898048 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 % _timestamp‚1.7885256412244103e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚122683679021568 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0èªBåP2º- _wandbruntime‚4879  _runtime‚48792M2Z»Ô /loss/FRt2/detection_loss‚3.0396906897939484 /loss/FRt4/detection_loss‚2.2417915864868765 1loss/SRqqt2/detection_loss‚2.4428305063928875 .loss/FBt1/detection_loss‚2.463501510412797 /loss/FBt2/detection_loss‚2.7237843149586727 2loss/SRqqt2/assignment_loss‚1.9617045983672141 2loss/SRqqt3/assignment_loss‚1.1202026418410242 /loss/FRt3/detection_loss‚1.4631185729169485 0loss/FBt1/assignment_loss‚0.3523151936370477 .loss/FBt3/detection_loss‚3.215622557534112 2loss/FBt3/assignment_loss‚0.013388339553785045 /loss/FRt1/assignment_loss‚3.424518037486721 /loss/FRt3/assignment_loss‚1.779184473283363 0loss/FRt4/assignment_loss‚2.5595237368887123 1loss/SRqqt1/assignment_loss‚1.619574637575583 0loss/FBt2/assignment_loss‚0.1663507958764967 1loss/SRqqt3/detection_loss‚0.9819152684739003 1loss/SRqqt4/assignment_loss‚2.390769934654236 1loss/SRqqt1/detection_loss‚1.9631096747788517 1loss/SRqqt4/detection_loss‚3.3219974954922993 %loss/total_loss‚47.47327488407995 % _timestamp‚1.7885256412287614e+09 1loss/FBt4/assignment_loss‚0.17625760458116613 /loss/FRt2/assignment_loss‚4.623315712501263 epoch‚2 .loss/FBt4/detection_loss‚0.876318038856754 trainer/global_step‚2849 _runtime‚4879.450395185 _step‚233 /loss/FRt1/detection_loss‚2.5524889617352873˜ `_¼Ù /loss/FRt2/assignment_loss‚4.623315712501263 epoch‚2 .loss/FBt4/detection_loss‚0.876318038856754 trainer/global_step‚2849 _runtime‚4879.450395185 _step‚233 /loss/FRt1/detection_loss‚2.5524889617352873 /loss/FRt2/detection_loss‚3.0396906897939484 /loss/FRt4/detection_loss‚2.2417915864868765 1loss/SRqqt2/detection_loss‚2.4428305063928875 .loss/FBt1/detection_loss‚2.463501510412797 /loss/FBt2/detection_loss‚2.7237843149586727 2loss/SRqqt2/assignment_loss‚1.9617045983672141 2loss/SRqqt3/assignment_loss‚1.1202026418410242 /loss/FRt3/detection_loss‚1.4631185729169485 0loss/FBt1/assignment_loss‚0.3523151936370477 .loss/FBt3/detection_loss‚3.215622557534112 2loss/FBt3/assignment_loss‚0.013388339553785045 /loss/FRt1/assignment_loss‚3.424518037486721 /loss/FRt3/assignment_loss‚1.779184473283363 0loss/FRt4/assignment_loss‚2.5595237368887123 1loss/SRqqt1/assignment_loss‚1.619574637575583 0loss/FBt2/assignment_loss‚0.1663507958764967 1loss/SRqqt3/detection_loss‚0.9819152684739003 1loss/SRqqt4/assignment_loss‚2.390769934654236 1loss/SRqqt1/detection_loss‚1.9631096747788517 1loss/SRqqt4/detection_loss‚3.3219974954922993 %loss/total_loss‚47.47327488407995 % _timestamp‚1.7885256412287614e+09 1loss/FBt4/assignment_loss‚0.17625760458116613齓b‘½‚(:† ÓøêÔõ¼ùõ gpu.0.gpu‚23.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚105.192' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.0384 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ë²aº{¾‚(:ð ÓøêÔœ½²ø network.recv‚ 1.596348e+06# disk.nvme0c0n1.in‚ 11480.2578125( disk.nvme0c0n1.out‚90833.61376953125 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda.out‚ 140.24609375) disk./.usagePercent‚78.89698355129063 disk.drbd1260.out‚ 0.66796875 disk.dm-0.in‚ 11480.2578125 disk.sda.in‚0 disk.sda3.in‚0& proc.memory.availableMB‚ 176.578125* proc.memory.percent‚17.850303649902344 proc.cpu.threads‚17 network.sent‚ 7.050648e+06& disk.nvme0n1.out‚90833.61376953125 disk.dm-1.in‚0# disk.dm-0.out‚90833.61376953125# memory_percent‚98.9222526550293" disk./.usageGB‚1175.6220703125! disk.nvme0n1.in‚ 11480.2578125 disk.dm-1.out‚ 140.24609375 disk.sda2.out‚0 disk.sda3.out‚ 140.24609375 proc.memory.rssMB‚ 2924.59375 cpu‚25.367072677997506 disk.drbd1260.in‚0,ô Žx¿‚(:í âøêÔ«–Ó÷( proc.memory.availableMB‚ 176.57421875 proc.cpu.threads‚17 network.sent‚ 7.144999e+06 disk.sda.in‚0 disk.sda.out‚ 140.6953125# disk.nvme0c0n1.in‚ 11480.2578125 disk.sda1.in‚0 disk.sda1.out‚0# memory_percent‚98.9222764968872 proc.memory.rssMB‚ 2924.59375) disk./.usagePercent‚78.89696742281598 disk.drbd1260.in‚0" disk.dm-0.out‚91022.2861328125 disk.sda3.in‚0! disk.nvme0n1.in‚ 11480.2578125 disk.dm-1.in‚0* proc.memory.percent‚17.850303649902344 network.recv‚ 1.601477e+06 disk.sda2.out‚0 disk.dm-0.in‚ 11480.2578125' disk.nvme0c0n1.out‚91022.2861328125 disk.dm-1.out‚ 140.6953125 cpu‚25.366726418080216% disk./.usageGB‚1175.6218299865723 disk.drbd1260.out‚0.671875 disk.sda2.in‚0 disk.sda3.out‚ 140.6953125% disk.nvme0n1.out‚91022.2861328125ŒoŽÀ‚(:ƒ âøêÔ쀡ø gpu.0.gpu‚12.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.14' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.628 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Š~6›Á‚(: ñøêÔô’Óö gpu.0.gpu‚33.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚95.972' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚19.194399999999998 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ô'»É|‚(:ñ ñøêÔÚÓÿ÷ proc.cpu.threads‚17% disk./.usageGB‚1176.2445030212402 disk.sda1.in‚0 disk.sda2.in‚0 disk.dm-0.in‚ 11480.2578125# disk.nvme0c0n1.in‚ 11480.2578125 proc.memory.rssMB‚ 2924.59375 cpu‚25.366320183538733) disk./.usagePercent‚78.93875553263003 disk.sda1.out‚0 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚0.671875 disk.sda2.out‚0 disk.sda.out‚ 140.95703125$ memory_percent‚98.92232418060303! disk.nvme0n1.in‚ 11480.2578125% disk.nvme0n1.out‚91352.3486328125 disk.dm-1.out‚ 140.95703125 disk.sda3.in‚0 disk.sda3.out‚ 140.95703125' disk.nvme0c0n1.out‚91352.3486328125( proc.memory.availableMB‚ 176.56640625 network.sent‚ 7.150782e+06 network.recv‚ 1.605461e+06" disk.dm-0.out‚91352.3486328125 disk.sda.in‚0* proc.memory.percent‚17.850303649902344»0‘Â(:† €ùêÔ³å÷ gpu.0.gpu‚17.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚101.848' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.3696 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Mx®¾}Ä‚(:ò €ùêÔ‘§Êø cpu‚25.365913990513135) disk./.usagePercent‚78.94671685492983& disk.nvme0n1.out‚91774.90478515625 disk.sda1.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 141.30078125 disk.sda2.out‚0# disk.dm-0.out‚91774.90478515625 disk.sda3.in‚0 network.sent‚ 7.156436e+06 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.sda.in‚0$ memory_percent‚98.91929626464844 proc.cpu.threads‚17 network.recv‚ 1.610364e+06! disk.nvme0n1.in‚ 11480.2578125 disk.sda1.in‚0 disk.dm-0.in‚ 11480.2578125 disk.sda.out‚ 141.30078125# disk.nvme0c0n1.in‚ 11480.2578125( disk.nvme0c0n1.out‚91774.90478515625$ proc.memory.availableMB‚177.0625% disk./.usageGB‚1176.3631324768066 disk.drbd1260.out‚ 0.67578125 disk.sda3.out‚ 141.30078125 proc.memory.rssMB‚ 2924.59375* proc.memory.percent‚17.850303649902344gõîÅ‚(:… ùêÔß¾“÷ gpu.0.gpu‚16.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.277' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.6554 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0kÒd\3Æ‚(:à ùêÔìâ¡ù proc.cpu.threads‚17 network.sent‚ 7.161304e+06 network.recv‚ 1.613631e+06" disk.nvme0n1.in‚11508.58984375! disk.nvme0n1.out‚ 92135.703125 disk.sda2.in‚0 disk.sda.out‚ 141.65625 proc.memory.rssMB‚ 2924.59375* proc.memory.percent‚17.8503036499023N"þ844) disk./.usagePercent‚78.94671429485449$ disk./.usageGB‚1176.363094329834 disk.sda3.in‚0# disk.nvme0c0n1.out‚ 92135.703125 disk.dm-1.in‚0 disk.drbd1260.in‚0$ memory_percent‚98.92199039459229( proc.memory.availableMB‚ 176.62109375 disk.sda3.out‚ 141.65625$ disk.nvme0c0n1.in‚11508.58984375 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-0.in‚11508.58984375 disk.dm-0.out‚ 92135.703125 disk.sda.in‚0 cpu‚25.365559110373304 disk.dm-1.out‚ 141.65625 disk.drbd1260.out‚ 0.67578125 disk.sda2.out‚0qM_P2Ç- _wandbruntime‚4961  _runtime‚4961v“þÅÈ¿ % lr-AdamW/pg1‚0.000865264282657151 % lr-AdamW/pg2‚0.000865264282657151 trainer/global_step‚2899 % _timestamp‚1.7885257235652823e+09 _runtime‚4961.785626406 _step‚234Ö©VÊÉÄ _step‚234 % lr-AdamW/pg1‚0.000865264282657151 % lr-AdamW/pg2‚0.000865264282657151 trainer/global_step‚2899 % _timestamp‚1.7885257235652823e+09 _runtime‚4961.785626406êó¹.2Ê- _wandbruntime‚4961  _runtime‚4961:ÿBÜ'ËÖO @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 _runtime‚4961.789515632 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚78715316 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚54948123 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 133663437 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚53674668 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚235 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚123490695004672 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚695 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚112072158364432 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5739097143503 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚124726193553920 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚78741659 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32806450 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚117811009487495 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚123490695004672 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚117735516037120 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5755412635648 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚130581012530688 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚117735534040064 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚117811255507935 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5854795036672 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚78741659 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚53673971 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 310365696 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚130580702164992 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚78741659 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32806450 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚117811009487495 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚123490695004672 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚117735516037120 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5755412635648 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚130581012530688 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚117735534040064 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚117811255507935 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5854795036672 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚78741659 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚53673971 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 310365696 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚130580702164992 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0ë˜\™o2Í- _wandbruntime‚4964  _runtime‚4964æÿµ=æ&ÎàM WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚117774802074112 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5757142993408 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53691728 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚78768377 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚117774820077056 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5757376674816 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚54966396 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 % _timestamp‚1.7885257259936047e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚123531945067520 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚112109688721856 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚54966396 trainer/global_step‚2899 _step‚236 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚78742008 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310352384 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚112109671682496 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 _runtime‚4964.219659759 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚130624423064576 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚117850498333905 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚117774802074112 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚130624733416960 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32817252 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚124767901464064 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚124767877523968 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚78768377 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20875178 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚54966398 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5741055632697 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚54966398 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286412288 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚53692432 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚702 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5856545540608 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚123532196751872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5740826651409 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 133708404 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚117774820077056 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20874476 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚704 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚123532196751872 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 133708404 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5856831952896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5757376674816 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚78742008 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5757142993408 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚117850744354553 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 1337347750…•ºÏåM FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 133734775 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚117774802074112 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5757142993408 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53691728 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚78768377 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚117774820077056 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5757376674816 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 NADeviceStatsMonitor.on_train_batch_end/alloca¥Ad,1"tion.large_pool.freed‚54966396 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 % _timestamp‚1.7885257259936047e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚123531945067520 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚112109688721856 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚54966396 trainer/global_step‚2899 _step‚236 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚78742008 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310352384 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚112109671682496 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 _runtime‚4964.219659759 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚130624423064576 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚117850498333905 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚117774802074112 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚130624733416960 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32817252 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚124767901464064 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚124767877523968 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚78768377 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20875178 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚54966398 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5741055632697 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚54966398 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286412288 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚53692432 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚702 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5856545540608 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚123532196751872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5740826651409 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 133708404 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚117774820077056 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20874476 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚704 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚123532196751872 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 133708404 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5856831952896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5757376674816 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚78742008 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5757142993408 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚117850744354553ìgtj‰2Ð- _wandbruntime‚4964  _runtime‚4964ȵê¨YÑÓ /loss/FRt4/assignment_loss‚2.829829994904793 1loss/FBt3/assignment_loss‚0.09981373090613488 0loss/SRqqt1/detection_loss‚2.020640432089567 /loss/FBt1/detection_loss‚2.5107720841964087 %loss/total_loss‚45.96299770763184 1loss/SRqqt4/detection_loss‚2.7033617371007015 /loss/FBt2/detection_loss‚2.4771781973540783 _runtime‚4964.224508232 0loss/SRqqt2/assignment_loss‚2.08354355980243 2loss/SRqqt3/assignment_loss‚1.1689750231922633 _step‚237 2loss/SRqqt4/assignment_loss‚1.6358214459921185 1loss/SRqqt2/detection_loss‚2.4076306990214755 /loss/FBt4/detection_loss‚0.8891765066039037 % _timestamp‚1.7885257260019717e+09 trainer/global_step‚2899 /loss/FRt3/assignment_loss‚2.028201087576444 .loss/FRt2/detection_loss‚3.210656056801478 1loss/SRqqt3/detection_loss‚1.0066140261028083 .loss/FRt1/assignment_loss‚3.29227314889431 1loss/FBt4/assignment_loss‚0.14040120856107557 /loss/FRt1/detection_loss‚3.2388907248775163 /loss/FRt3/detection_loss‚1.9868450880539221 /loss/FRt4/detection_loss‚1.7668143153383982 /loss/FRt2/assignment_loss‚4.733824511369069 1loss/FBt2/assignment_loss‚0.04843260934322302 epoch‚2 2loss/SRqqt1/assignment_loss‚1.3590923290699721 0loss/FBt1/assignment_loss‚0.2707006629028304 /loss/FBt3/detection_loss‚2.0535085275769234Wþz^ÒØ /loss/FRt4/assignment_loss‚2.829829994904793 1loss/FBt3/assignment_loss‚0.09981373090613488 0loss/SRqqt1/detection_loss‚2.020640432089567 /loss/FBt1/detection_loss‚2.5107720841964087 %loss/total_loss‚45.96299770763184 1loss/SRqqt4/detection_loss‚2.7033617371007015 /loss/FBt2/detection_loss‚2.4771781973540783 _runtime‚4964.224508232 0loss/SRqqt2/assignment_loss‚2.08354355980243 2loss/SRqqt3/assignment_loss‚1.1689750231922633 _step‚237 2loss/SRqqt4/assignment_loss‚1.6358214459921185 1loss/SRqqt2/detection_loss‚2.4076306990214755 /loss/FBt4/detection_loss‚0.8891765066039037 % _timestamp‚1.7885257260019717e+09 trainer/global_step‚2899 /loss/FRt3/assignment_loss‚2.028201087576444 .loss/FRt2/detection_loss‚3.210656056801478 1loss/SRqqt3/detection_loss‚1.0066140261028083 .loss/FRt1/assignment_loss‚3.29227314889431 1loss/FBt4/assignment_loss‚0.14040120856107557 /loss/FRt1/detection_loss‚3.2388907248775163 /loss/FRt3/detection_loss‚1.9868450880539221 /loss/FRt4/detection_loss‚1.7668143153383982 /loss/FRt2/assignment_loss‚4.733824511369069 1loss/FBt2/assignment_loss‚0.04843260934322302 epoch‚2 2loss/SRqqt1/assignment_loss‚1.3590923290699721 0loss/FBt1/assignment_loss‚0.2707006629028304 /loss/FBt3/detection_loss‚2.0535085275769234í1ÃÏ|›Ó‚(: žùêÔ†€Øø gpu.0.gpu‚17.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚100.43' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚20.086000000000002 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0͈5d}Ô‚(:ò žùêÔÎÅù proc.memory.rssMB‚ 2924.59375 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.dm-0.in‚11612.33203125# disk.dm-0.out‚92435.10595703125 disk.sda.in‚0 cpu‚25.365314787168945 network.recv‚ 1.617284e+06% disk./.usageGB‚1176.3640403747559$ disk.nvme0c0n1.in‚11612.33203125( disk.nvme0c0n1.out‚92435.10595703125" disk.nvme0n1.in‚11612.33203125& disk.nvme0n1.out‚92435.10595703125$ memory_percent‚98.92375469207764( proc.memory.availableMB‚ 176.33203125) disk./.usagePercent‚78.94677778472298 disk.drbd1260.out‚ 0.6796875 disk.sda3.in‚0 disk.sda3.out‚ 142.109375* proc.memory.percent‚17.850303649902344 proc.cpu.threads‚17 network.sent‚ 7.166468e+06 disk.dm-1.out‚ 142.109375 disk.sda2.out‚0 disk.sda.out‚ 142.109375 disk.sda1.in‚0 disk.sda1.out‚0£Ô[ê€Õ‚(:õ ­ùêÔ裶÷# disk.dm-0.out‚92637.54345703125 disk.sda3.in‚0$ disk.nvme0c0n1.in‚11612.33203125" disk.nvme0n1.in‚11612.33203125( proc.memory.availableMB‚ 176.55078125* proc.memory.percent‚17.850303649902344% disk./.usageGB‚1176.3640403747559 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda2.out‚0 disk.sda.out‚ 142.4921875& disk.nvme0n1.out‚92637.54345703125 disk.dm-1.out‚ 142.4921875 disk.dm-0.in‚11612.33203125 disk.sda3.out‚ 142.4921875( disk.nvme0c0n1.out‚92637.54345703125$ memory_percent‚98.92241954803467 proc.memory.rssMB‚ 2924.59375 cpu‚25.365137766334595 proc.cpu.threads‚17 network.sent‚ 7.274692e+06 disk.drbd1260.out‚ 0.6796875 disk.sda2.in‚0 disk.sda.in‚0 network.recv‚ 1.623662e+06) disk./.usagePercent‚78.94677778472298ƒ/VšÖ‚(: ­ùêÔ¾Êß÷ gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚124.62' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.924000000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0(n…ׂ(:… ¼ùêÔ‰â¾ù gpu.0.gpu‚21.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚102.015' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.403 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0p³nƒØ‚(:ø ¼ùêÔÀü¸û disk.sda3.out‚ 142.90234375) disk./.usagePercent‚78.94748257346474# disk.dm-0.out‚92855.35205078125 disk.sda.out‚ 142.90234375$ disk.nvme0c0n1.in‚11612.33203125 disk.dm-1.in‚0' proc.memory.availableMB‚ 176.5546875* proc.memory.percent‚17.850303649902344% disk./.usageGB‚1176.3745422363281 disk.drbd1260.in‚0 disk.sda2.in‚0" disk.nvme0n1.in‚11612.33203125 disk.sda1.out‚0 disk.dm-1.out‚ 142.90234375$ memory_percent‚98.92239570617676 network.sent‚ 7.281103e+06 network.recv‚ 1.628644e+06 disk.sda2.out‚0 disk.sda.in‚0 disk.sda3.in‚0 disk.sda1.in‚0 proc.cpu.threads‚17 disk.drbd1260.out‚ 0.68359375 disk.dm-0.in‚11612.33203125( disk.nvme0c0n1.out‚92855.35205078125& disk.nvme0n1.out‚92855.35205078125 proc.memory.rssMB‚ 2924.59375 cpu‚25.364902159544084O‰uÙ‚(:ê ËùêÔèõ ø( disk.nvme0c0n1.out‚93129.91455078125 network.recv‚ 1.632205e+06 disk.sda2.in‚0 disk.dm-0.in‚11612.33203125 disk.sda.in‚0 disk.sda3.in‚0 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.68359375 disk.sda.out‚ 143.21875$ memory_percent‚98.92244338989258 proc.memory.rssMB‚ 2924.59375* proc.memory.percent‚17.850303649902344 cpu‚25.36459485598584 proc.cpu.threads‚17( disk./.usagePercent‚78.9459076151141# disk./.usageGB‚1176.35107421875 disk.sda3.out‚ 143.21875" disk.nvme0n1.in‚11612.33203125& disk.nvme0n1.out‚93129.91455078125 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.in‚0 network.sent‚ 7.286039e+06 disk.sda2.out‚0# disk.dm-0.out‚93129.91455078125$ disk.nvme0c0n1.in‚11612.33203125 disk.dm-1.out‚ 143.21875& proc.memory.availableMB‚ 176.546875Ü8"‘Ú‚(:† ËùêÔïåÜø gpu.0.gpu‚18.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚102.422' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.4844 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ôðÂÿZÛ‚(:Ï ÚùêÔ̦Ñ÷% disk.nvme0c0n1.out‚93438.71484375 disk.nvme0n1.in‚ 11770.234375 disk.sda1.in‚0 disk.dm-1.in‚0 disk.drbd1260.out‚0.6875 disk.sda.in‚0 network.recv‚ 1.635789e+06 disk.sda3.in‚0$ memory_percent‚98.92244338989258 proc.memory.rssMB‚ 2924.59375* proc.memory.percent‚17.850303649902344 cpu‚25.364213530281013 proc.cpu.threads‚17 network.sent‚ 7.291183e+06# disk./.usageGB‚1176.35107421875 disk.dm-1.out‚143.75 disk.drbd1260.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 11770.234375 disk.dm-0.out‚93438.71484375 disk.sda.out‚143.75( disk./.usagePercent‚78.9459076151141# disk.nvme0n1.out‚93438.71484375 disk.sda1.out‚0 disk.sda2.in‚0& proc.memory.availableMB‚ 176.546875 disk.sda3.out‚143.75" disk.nvme0c0n1.in‚ 11770.234375Ž7æœÜ‚(:‘ ÚùêÔÆÿÉ÷ gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚137.525' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.505000000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Z#‰úÝ‚(:… éùêÔçÿœ÷ gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚102.135' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.427 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0üßÕIhÞ‚(:Ý éùêÔ“Þú disk.sda1.out‚0 disk.dm-1.out‚ 144.03125 disk.sda.in‚0% disk.nvme0c0n1.out‚93837.44921875 proc.memory.rssMB‚ 2924.59375) disk./.usagePercent‚78.91834661200039 disk.sda2.in‚0" disk.nvme0c0n1.in‚ 11770.234375( proc.memory.availableMB‚ 176.04296875 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚0.6875 disk.sda2.out‚0 disk.dm-0.in‚ 11770.234375 disk.dm-0.out‚93837.44921875 disk.sda3.in‚0 disk.sda3.out‚ 144.03125% disk./.usageGB‚1175.9403953552246 disk.sda.out‚ 144.03125 disk.nvme0n1.in‚ 11770.234375# disk.nvme0n1.out‚93837.44921875$ memory_percent‚98.92551898956299* proc.memory.percent‚17.850303649902344 cpu‚25.363992223414566 proc.cpu.threads‚17 network.sent‚ 7.297029e+06 network.recv‚ 1.640274e+06 disk.sda1.in‚0“5€®2ß- _wandbruntime‚5049  _runtime‚5049b‡ÀfÇàÁ & lr-AdamW/pg1‚0.0008621127085290675 & lr-AdamW/pg2‚0.0008621127085290675 trainer/global_step‚2949 % _timestamp‚1.7885258113881283e+09 _runtime‚5049.607730419 _step‚238®å0äÌáÆ & lr-AdamW/pg1‚0.0008621127085290675 & lr-AdamW/pg2‚0.0008621127085290675 trainer/global_step‚2949 % _timestamp‚1.7885258113881283e+09 _runtime‚5049.607730419 _step‚238î×£}¢2â- _wandbruntime‚5049  _runtime‚5049AüÿÜ'ãÖO =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚80079611 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33346552 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚125553212560896 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚119699835889664 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚119699835889664 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚80053268 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5954753702400 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5853628342272 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚126810392056320 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚125553464231936 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚125553212560896 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚119699817886720 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21215293 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚113948659196272 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚55861871 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚119785465399203 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚5049.615080789 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21214627 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚80053268 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5853394674176 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚55861871 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚132764861430272 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚239400969øRÄ@ äÛO KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚80053268 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5954753702400 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5853628342272 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚126810392056320 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚125553464231936 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚125553212560896 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚119699817886720 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21215293 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚113948659196272 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚55861871 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚119785465399203 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.cóÚà¡small_pool.freed‚0 _runtime‚5049.615080789 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21214627 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚80053268 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5853394674176 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚55861871 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚132764861430272 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚80079611 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33346552 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚125553212560896 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚119699835889664 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚119699835889664 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773ïsŽÖ­2å- _wandbruntime‚5051  _runtime‚5051¿•Cæ&æàM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 135986477 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306158080 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚132808541869568 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 trainer/global_step‚2949 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 135960106 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 _step‚240 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚125594462623744 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217984 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5956748061184 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚119825200266261 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚55880146 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚5051.272182296 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚125594714308096 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5855592381440 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚119824954245613 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚119739103923712 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚694 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚119739121926656 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚126852076026368 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33357354 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5956465843200 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 135986477 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚80106329 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54578917 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚80079960 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21222259 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5838993673205 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚119739103923712 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚55880146 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5855592381440 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54579613 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚119739121926656 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5838764691917 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33357352 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21221565 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚132808848027648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0ýýë&çåM REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33357352 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21221565 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚132808848027648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 135986477 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306158080 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚132808541869568 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 trainer/global_step‚2949 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 135960106 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 _step‚240 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚125594462623744 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217984 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5956748061184 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚119825200266261 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚55880146 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚5051.272182296 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚125594714308096 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5855592381440 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚119824954245613 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚119739103923712 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚694 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚119739121926656 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚126852076026368 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33357354 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5956465843200 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 135986477 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚80106329 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54578917 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚80079960 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21222259 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5838993673205 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚119739103923712 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚55880146 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5855592381440 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54579613 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚119739121926656 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5838764691917 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0ðñzšÓ2è- _wandbruntime‚5051  _runtime‚5051¡†8¡WéÑ epoch‚2 2loss/SRqqt2/assignment_loss‚1.8261745890070284 2loss/SRqqt4/assignment_loss‚2.0612724040235793 1loss/SRqqt3/detection_loss‚0.8925275518285001 trainer/global_step‚2949 -loss/FRt2/assignment_loss‚4.2018502046322 2loss/SRqqt1/assignment_loss‚1.4233900593593716 0loss/FBt1/assignment_loss‚0.3220981351045008 % _timestamp‚1.7885258130549564e+09 1loss/FBt3/assignment_loss‚0.14350965529385057 1loss/FBt4/assignment_loss‚0.20290026555442506 .loss/FRt1/detection_loss‚2.918515294790268 _runtime‚5051.274363265 /loss/FRt3/detection_loss‚1.8191680740891842 /loss/FRt4/detection_loss‚2.3749742619011824 /loss/FBt4/detection_loss‚0.9071696127232697 1loss/SRqqt1/detection_loss‚2.2537655189633368 /loss/FBt1/detection_loss‚1.8978525075045498 ,loss/FRt2/detection_loss‚2.9590643336033 /loss/SRqqt2/detection_loss‚2.56307185121945 _step‚241 /loss/FRt4/assignment_loss‚2.891177194874461 1loss/SRqqt4/detection_loss‚3.0134396872350147 /loss/FBt2/detection_loss‚3.0380015770594277 /loss/FRt1/assignment_loss‚3.331497999338003 0loss/FRt3/assignment_loss‚1.3151126117037053 2loss/SRqqt3/assignment_loss‚0.9921677422209862 1loss/FBt2/assignment_loss‚0.26435127068543807 /loss/FBt3/detection_loss‚2.2226553772177016 %loss/total_loss‚45.83570777993274A´k\êÖ /loss/FRt1/assignment_loss‚3.331497999338003 0loss/FRt3/assignment_loss‚1.3151126117037053 2loss/SRqqt3/assignment_loss‚0.9921677422209862 1loss/FBt2/assignment_loss‚0.26435127068543807 /loss/FBt3/detection_loss‚2.2226553772177016 %loss/total_loss‚45.83570777993274 epoch‚2 2loss/SRqqt2/assignment_loss‚1.8261745890070284 2loss/SRqqt4/assignment_loss‚2.0612724040235793 1loss/SRqqt3/detection_loss‚0.8925275518285001 trainer/global_step‚2949 -loss/FRt2/assignment_loss‚4.2018502046322 2loss/SRqqt1/assignment_loss‚1.4233900593593716 0loss/FBt1/assignment_loss‚0.3220981351045008 % _timestamp‚1.7885258130549564e+09 1loss/FBt3/assignment_loss‚0.14350965529385057 1loss/FBt4/assignment_loss‚0.20290026555442506 .loss/FRt1/detection_loss‚2.918515294790268 _runtime‚5051.274363265 /loss/FRt3/detection_loss‚1.8191680740891842 /loss/FRt4/detection_loss‚2.3749742619011824 /loss/FBt4/detection_loss‚0.9071696127232697 1loss/SRqqt1/detection_loss‚2.2537655189633368 /loss/FBt1/detection_loss‚1.8978525075045498 ,loss/FRt2/detection_loss‚2.9590643336033 /loss/SRqqt2/detection_loss‚2.56307185121945 _step‚241 /loss/FRt4/assignment_loss‚2.891177194874461 1loss/SRqqt4/detection_loss‚3.0134396872350147 /loss/FBt2/detection_loss‚3.0380015770594277ñï¸+pë‚(:å øùêÔùÀ˜ø) disk./.usagePercent‚78.92676593177896 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚0' disk.nvme0c0n1.out‚94088.4189453125 disk.nvme0n1.in‚ 11770.234375% disk.nvme0n1.out‚94088.4189453125 disk.dm-1.out‚144.4375" disk.dm-0.out‚94088.4189453125 disk.sda.in‚0 disk.sda3.out‚144.4375 disk.dm-1.in‚0 disk.drbd1260.in‚0$ memory_percent‚98.39053153991699' proc.memory.availableMB‚ 263.6953125# proc.memory.rssMB‚ 2833.52734375 network.sent‚ 7.303242e+06 disk.dm-0.in‚ 11770.234375 disk.drbd1260.out‚ 0.69140625 proc.cpu.threads‚17 network.recv‚ 1.645003e+06% disk./.usageGB‚1176.0658493041992 disk.sda.out‚144.4375" disk.nvme0c0n1.in‚ 11770.234375 disk.sda1.in‚0 disk.sda1.out‚0) proc.memory.percent‚17.29447841644287 cpu‚25.363683572894136'a´ì‚(:„ øùêÔêÈù gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚99.733' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.9466 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©0¸ž|í‚(:ñ ‡úêÔ×ÈÈ÷ disk.sda2.in‚0 disk.sda3.out‚ 144.80859375# proc.memory.rssMB‚ 2833.52734375 cpu‚25.363405802801665 proc.cpu.threads‚17) disk./.usagePercent‚78.92673137076183% disk.nvme0n1.out‚94295.1376953125 disk.sda1.out‚0 disk.drbd1260.in‚0" disk.dm-0.out‚94295.1376953125 disk.sda.out‚ 144.80859375 disk.sda1.in‚0 disk.dm-1.out‚ 144.80859375 disk.dm-0.in‚ 11770.234375 disk.sda.in‚0 disk.sda3.in‚0$ memory_percent‚98.39155673980713( proc.memory.availableMB‚ 263.52734375) proc.memory.percent‚17.29447841644287 network.sent‚ 7.410774e+06 network.recv‚ 1.65021e+06% disk./.usageGB‚1176.0653343200684" disk.nvme0c0n1.in‚ 11770.234375' disk.nvme0c0n1.out‚94295.1376953125 disk.nvme0n1.in‚ 11770.234375 disk.dm-1.in‚0 disk.sda2.out‚0 disk.drbd1260.out‚ 0.69140625Sð†ï¯î‚(:† ‡úêÔõ¤Ëù gpu.0.gpu‚18.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0LäÙ'â gpu.0.powerWatts‚100.313' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.0626 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0($ï‚(:… –úêÔÀ‚¦÷ gpu.0.gpu‚31.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚123.325' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.665 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0KüØxð‚(:í –úêÔ‚¼óú$ disk.nvme0c0n1.out‚ 94718.4609375 disk.sda1.in‚0$ memory_percent‚98.39024543762207 disk.sda2.in‚0 disk.sda3.in‚0" disk.nvme0n1.in‚11947.71484375" disk.nvme0n1.out‚ 94718.4609375 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.6953125' proc.memory.availableMB‚ 263.7421875% disk./.usageGB‚1175.5632858276367 disk.sda2.out‚0 disk.sda3.out‚ 145.30078125 disk.sda1.out‚0# proc.memory.rssMB‚ 2833.52734375) proc.memory.percent‚17.29447841644287 network.sent‚ 7.416061e+06 disk.sda.in‚0$ disk.nvme0c0n1.in‚11947.71484375 disk.dm-1.out‚ 145.30078125 cpu‚25.362797106576377 proc.cpu.threads‚17 network.recv‚ 1.653721e+06) disk./.usagePercent‚78.89303847518799 disk.dm-0.in‚11947.71484375 disk.dm-0.out‚ 94718.4609375 disk.sda.out‚ 145.30078125,|!T]ñ‚(:Ò ¥úêÔ©é½÷) disk./.usagePercent‚78.89297242524415% disk./.usageGB‚1175.5623016357422 disk.drbd1260.out‚ 0.6953125$ disk.nvme0c0n1.in‚11947.71484375 disk.sda1.in‚0) proc.memory.percent‚17.29447841644287 disk.drbd1260.in‚0 disk.sda2.out‚0 disk.dm-0.out‚95075 disk.sda1.out‚0$ memory_percent‚98.39177131652832# proc.memory.rssMB‚ 2833.52734375 network.recv‚ 1.658332e+06 disk.dm-1.out‚ 145.7578125 disk.sda.out‚ 145.7578125 disk.sda3.in‚0 disk.nvme0c0n1.out‚95075 disk.nvme0n1.out‚95075 cpu‚25.361969727536056 proc.cpu.threads‚17 network.sent‚ 7.421914e+06 disk.dm-1.in‚0 disk.sda2.in‚0 disk.dm-0.in‚11947.71484375 disk.sda.in‚0 disk.sda3.out‚ 145.7578125" disk.nvme0n1.in‚11947.71484375' proc.memory.availableMB‚ 263.4921875Kðc·œò‚(:‘ ¥úêÔí©€ø gpu.0.gpu‚30.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚33.0 gpu.0.powerWatts‚140.466' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.093200000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0®s$À2ó- _wandbruntime‚5111  _runtime‚5111– Ö©ÆôÀ % _timestamp‚1.7885258732420595e+09 _runtime‚ 5111.46159484 _step‚242 & lr-AdamW/pg1‚0.0008588772181269154 & lr-AdamW/pg2‚0.0008588772181269154 trainer/global_step‚2999düã°ËõÅ _runtime‚ 5111.46159484 _step‚242 & lr-AdamW/pg1‚0.0008588772181269154 & lr-AdamW/pg2‚0.0008588772181269154 trainer/global_step‚2999 % _timestamp‚1.7885258732420595e+09ò2ê2ö- _wandbruntime‚5111  _runtime‚5111´öµ‚Ü'÷ÖO JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚704 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33886652 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268544 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚121664119736320 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 138193186 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33886650 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚81391220 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚127615981788160 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 _runtime‚5111.462807865 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21560538 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚121759921310911 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚134948990576128 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚128894614498816 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚115825177067472 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚121664137739264 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6054684345856 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6054400017408 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 % _timestamp‚1.7885258732432091e+09 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21559836 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚127615730117120 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5951844048896 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚56775623 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚81417563 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚121664137739264 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚127615730117120 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5951844048896 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚56775623 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚81417563 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚121664137739264 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268544 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚121664119736320 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 138193186 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33886650 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚81391220 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚127615981788160 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 _runtime‚5111.462807865 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21560538 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚121759921310911 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚134948990576128 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚128894614498816 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚115825177067472 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚121664137739264 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6054684345856 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6054400017408 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 % _timestamp‚1.7885258732432091e+09 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21559836 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚56793896 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚81417912 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33897454 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚134992982080512 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 327129600 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5936702732425 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 138211808 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5936931713713 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5953808088064 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6056683611648 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚81417912 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 303189504 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚127657231864320 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚55464650 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚115862724464256 % _timestamp‚1.7885258741943042e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚81444281 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚121703423776256 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚128936274528768 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚121703405773312 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚121703405773312 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚115862707424896 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55463969 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚121799656177969 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚121799410157321 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 138211808 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 B8DeviceStatsMonitor.on_train_batch_end/active.ò·ùÌall.current‚26371 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚121703423776256 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 138238179 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚2999 _step‚244 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5953574406656 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚681 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496~¤ê&ûäM SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 138211808 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚121703423776256 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 138238179 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚2999 _step‚244 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5953574406656 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚681 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 138238179 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6056380422144 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚127656980179968 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚56793896 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚81417912 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33897454 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚134992982080512 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 327129600 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5936702732425 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 138211808 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5936931713713 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5953808088064 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6056683611648 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚81417912 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 303189504 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚127657231864320 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚55464650 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚115862724464256 % _timestamp‚1.7885258741943042e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚81444281 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚121703423776256 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚128936274528768 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚121703405773312 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚121703405773312 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚115862707424896 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55463969 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚121799656177969 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚121799410157321ô;à M2ü- _wandbruntime‚5112  _runtime‚5112ûÙr‘^ýØ 1loss/SRqqt3/detection_loss‚1.0661682557118566 2loss/SRqqt3/assignment_loss‚1.2399670703810428 1loss/SRqqt1/detection_loss‚2.0409964842999235 /loss/FBt2/detection_loss‚2.1149430414040884 epoch‚2 % _timestamp‚1.7885258741966617e+09 _step‚245 0loss/FRt4/assignment_loss‚2.7830912184715273 2loss/SRqqt2/assignment_loss‚2.0427223336080025 /loss/FBt3/detection_loss‚2.6194445848464967 trainer/global_step‚2999 2loss/SRqqt4/assignment_loss‚2.4370871081071743 /loss/FBt4/detection_loss‚0.8134935557677904 2loss/SRqqt1/assignment_loss‚1.6989168454595702 /loss/FRt4/detection_loss‚1.7143637480353937 &loss/total_loss‚45.848824078020314 _runtime‚5112.415968211 /loss/FRt1/assignment_loss‚3.753541773557663 0loss/FRt3/assignment_loss‚1.6233854658568083 0loss/FBt4/assignment_loss‚0.1303483339974772 /loss/FRt3/detection_loss‚1.6789184858772292 1loss/SRqqt4/detection_loss‚2.5718124053057503 .loss/FBt1/detection_loss‚2.370463863015175 0loss/FRt2/assignment_loss‚3.3478601276874542 0loss/FBt1/assignment_loss‚0.3019437861552693 1loss/FBt2/assignment_loss‚0.30903600093630906 0loss/FBt3/assignment_loss‚0.1684898402541876 /loss/FRt1/detection_loss‚2.9959258437156677 .loss/FRt2/detection_loss‚3.244829735159874 1loss/SRqqt2/detection_loss‚2.7810741704085777RDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21906885 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚129678499344384 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚56333635 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 trainer/global_step‚3049 _runtime‚5174.911870155 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6032682283947 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚57689371 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6050059755520 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6049826087424 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268544 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚57689373 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚56334329 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284328448 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚130978813001216 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 % _timestamp‚1.7885259366917648e+09 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚123628421585920 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚82729172 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6049826087424 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34426752 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚82755515 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34426750 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚123628439588864 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚57689371 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚82729172 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚692 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚129678499344384 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6154370690048 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6154655018496 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚129678247673344 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚694 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚137133468019712 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚129678247673344 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚117701711978032 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚117701694938672 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6032911265027 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚123628421585920 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚123734623243059ðö=Üš ŒÛO LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚130978813001216 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 % _timestamp‚1.7885259366917648e+09 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚123628421585920 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚82729172 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6049826087424 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34426752 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚82755515 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34426750 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚123628439588864 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚57689371 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚82729172 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚692 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚129678499344384 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6154370690048 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 H?DeviceStatsMonitor.on_trai}™,Gn_batch_start/inactive_split.all.peak‚1319 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6154655018496 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚129678247673344 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚694 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚137133468019712 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚129678247673344 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚117701711978032 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚117701694938672 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6032911265027 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚123628421585920 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚123734623243059 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚123628439588864 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6050059755520 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 140418543 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚137133159751168 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚123734377222619 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚57689373 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 140444888 _step‚247 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚130978789061120 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21906885 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚129678499344384 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚56333635 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 trainer/global_step‚3049 _runtime‚5174.911870155 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6032682283947 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚57689371 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6050059755520 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6049826087424 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268544 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚57689373 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚56334329 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284328448 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0÷á ¦´‘‚(:† ðúêÔýβ÷ gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚108.058' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.6116 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÄyÇxŽ‚(:í ðúêÔÄ“Ðú# disk.nvme0c0n1.in‚ 12094.0859375 disk.sda2.out‚0# proc.memory.rssMB‚ 2833.52734375 proc.cpu.threads‚17" disk.dm-0.out‚96352.5947265625% disk.nvme0n1.out‚96352.5947265625 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.70703125& proc.memory.availableMB‚ 263.359375 network.sent‚ 7.552802e+06 disk.sda.in‚0! disk.nvme0n1.in‚ 12094.0859375 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 148.1796875) proc.memory.percent‚17.29447841644287 cpu‚25.3589916521213) disk./.usagePercent‚78.91363684139296 disk.dm-0.in‚ 12094.0859375 disk.sda.out‚ 148.1796875 disk.sda3.in‚0' disk.nvme0c0n1.out‚96352.5947265625 disk.sda2.in‚0$ memory_percent‚98.39258193969727 network.recv‚ 1.681249e+06$ disk./.usageGB‚1175.870216369629 disk.sda3.out‚ 148.1796875л˜2- _wandbruntime‚5175  _runtime‚5175:+¦æ&àM MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚117739242335456 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6034869754221 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 275926528 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚57707648 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚667 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56351496 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚57707646 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 _runtime‚5175.949796305 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚82782233 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚123774112089677 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 299866624 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚82755864 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 140489881 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚123667725625856 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 140489881 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚123667725625856 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚57707648 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6156651231232 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6156375304704 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚123667707622912 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚665 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6034640772933 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34437552 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚129719749420544 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚123773866069029 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 trainer/global_step‚3049 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21913944 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚117739225296096 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 % _timestamp‚1.7885259377269914e+09 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34437554 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚129719497736192 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚137177148202496 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚137176848335872 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚57707648 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚667 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56351496 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚57707646 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 _runtime‚5175.949796305 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚82782233 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚123774112089677 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 299866624 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚82755864 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 140489881 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚123667725625856 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 140489881 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚123667725625856 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚57707648 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6156651231232 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6156375304704 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚123667707622912 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚665 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6034640772933 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34437552 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚129719749420544 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚123773866069029 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 trainer/global_step‚3049 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21913944 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚117739225296096 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 % _timestamp‚1.7885259377269914e+09 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34437554 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚129719497736192 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚137177148202496 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚137176848335872 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608øíøH~2’- _wandbruntime‚5175  _runtime‚5175uv°õ[“Õ 2loss/SRqqt3/assignment_loss‚0.9622780807734093 epoch‚2 trainer/global_step‚3049 0loss/FRt1/assignment_loss‚3.0752923751578614 1loss/FBt4/assignment_loss‚0.24195681907091854 /loss/FBt2/detection_loss‚2.6878017046872307 .loss/FBt3/detection_loss‚3.457065840562185 _runtime‚5175.951855305 0loss/FRt3/assignment_loss‚1.9163745085398356 1loss/SRqqt4/assignment_loss‚2.409415729343891 /loss/FRt4/detection_loss‚1.6616400213518108 0loss/SRqqt3/detection_loss‚1.112881231791923 /loss/FBt1/detection_loss‚2.4202732619117286 /loss/FBt4/detection_loss‚0.8256796767624716 /loss/FRt2/assignment_loss‚4.183514182766278 0loss/FRt4/assignment_loss‚2.8311144342789283 % _timestamp‚1.7885259377304246e+09 0loss/SRqqt1/detection_loss‚2.168052287772298 /loss/FRt2/detection_loss‚3.1606834630171456 2loss/FBt3/assignment_loss‚0.004213759520401557 .loss/FRt1/detection_loss‚2.827336319229182 /loss/FRt3/detection_loss‚1.6457464319467545 0loss/SRqqt4/detection_loss‚2.560938846319914 _step‚249 1loss/FBt1/assignment_loss‚0.19425994259055349 1loss/FBt2/assignment_loss‚0.12516713714633035 1loss/SRqqt1/assignment_loss‚1.453500943025574 1loss/SRqqt2/assignment_loss‚2.099274802953005 1loss/SRqqt2/detection_loss‚2.9559001363813877 %loss/total_loss‚46.98036193690102ø3`”Ú 1loss/SRqqt2/detection_loss‚2.9559001363813877 %loss/total_loss‚46.98036193690102 2loss/SRqqt3/assignment_loss‚0.9622780807734093 epoch‚2 trainer/global_step‚3049 0loss/FRt1/assignment_loss‚3.0752923751578614 1loss/FBt4/assignment_loss‚0.24195681907091854 /loss/FBt2/detection_loss‚2.6878017046872307 .loss/FBt3/detection_loss‚3.457065840562185 _runtime‚5175.951855305 0loss/FRt3/assignment_loss‚1.9163745085398356 1loss/SRqqt4/assignment_loss‚2.409415729343891 /loss/FRt4/detection_loss‚1.6616400213518108 0loss/SRqqt3/detection_loss‚1.112881231791923 /loss/FBt1/detection_loss‚2.4202732619117286 /loss/FBt4/detection_loss‚0.8256796767624716 /loss/FRt2/assignment_loss‚4.183514182766278 0loss/FRt4/assignment_loss‚2.8311144342789283 % _timestamp‚1.7885259377304246e+09 0loss/SRqqt1/detection_loss‚2.168052287772298 /loss/FRt2/detection_loss‚3.1606834630171456 2loss/FBt3/assignment_loss‚0.004213759520401557 .loss/FRt1/detection_loss‚2.827336319229182 /loss/FRt3/detection_loss‚1.6457464319467545 0loss/SRqqt4/detection_loss‚2.560938846319914 _step‚249 1loss/FBt1/assignment_loss‚0.19425994259055349 1loss/FBt2/assignment_loss‚0.12516713714633035 1loss/SRqqt1/assignment_loss‚1.453500943025574 1loss/SRqqt2/assignment_loss‚2.099274802953005ùp÷Zì•‚(:‚ ÿúêÔË Ìø gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.14' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.628 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0?ÕÕ¡~–‚(:ó ÿúêÔµÖù disk.dm-1.out‚ 148.62109375 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.70703125 network.sent‚ 7.648865e+06 network.recv‚ 1.686654e+06 disk.sda2.out‚0 disk.dm-0.in‚ 12094.0859375! disk.nvme0n1.in‚ 12094.0859375 disk.sda1.in‚0$ memory_percent‚98.38442802429199' proc.memory.availableMB‚ 264.6953125! proc.memory.rssMB‚ 2836.296875* proc.memory.percent‚17.311382293701172 cpu‚25.358742781967077 proc.cpu.threads‚17 disk.sda3.in‚0# disk.nvme0c0n1.in‚ 12094.0859375' disk.nvme0c0n1.out‚96716.3720703125) disk./.usagePercent‚78.94716998826543% disk./.usageGB‚1176.3698844909668" disk.dm-0.out‚96716.3720703125 disk.sda.in‚0% disk.nvme0n1.out‚96716.3720703125 disk.sda2.in‚0 disk.sda.out‚ 148.62109375 disk.sda3.out‚ 148.62109375 disk.sda1.out‚0 disk.dm-1.in‚0p®¡¨a—‚(:… ŽûêÔªØÞõ gpu.0.gpu‚26.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.95Åû( /0271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚93.277' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.6554 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ã¦C±„˜‚(:ù ŽûêÔ¾Àƒø( proc.memory.availableMB‚ 264.94140625! proc.memory.rssMB‚ 2836.296875 cpu‚25.358300484977075 proc.cpu.threads‚17( disk.nvme0c0n1.out‚97033.68017578125" disk.nvme0n1.in‚12261.62109375 disk.dm-1.out‚ 148.94140625 disk.drbd1260.in‚0 disk.sda2.out‚0 disk.sda.in‚0 disk.sda.out‚ 148.94140625 disk.sda3.in‚0& disk.nvme0n1.out‚97033.68017578125 disk.sda1.in‚0 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.7109375 disk.sda2.in‚0 disk.dm-0.in‚12261.62109375 disk.sda3.out‚ 148.94140625$ memory_percent‚98.38292598724365 network.recv‚ 1.692092e+06$ disk.nvme0c0n1.in‚12261.62109375 disk.sda1.out‚0* proc.memory.percent‚17.311382293701172 network.sent‚ 7.745776e+06) disk./.usagePercent‚78.94716998826543% disk./.usageGB‚1176.3698844909668# disk.dm-0.out‚97033.68017578125ˆ˜ †™‚(:… ûêÔ¯Ñ÷÷ gpu.0.gpu‚43.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚139.035' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.807 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0è°5š‚(:ö ûêÔ„¯Öø' proc.memory.availableMB‚ 264.4140625 disk.sda.in‚0 disk.sda3.in‚0( disk.nvme0c0n1.out‚97321.66845703125 disk.sda1.out‚0 disk.dm-1.out‚ 149.37890625 disk.drbd1260.in‚0$ memory_percent‚98.38614463806152 proc.cpu.threads‚17) disk./.usagePercent‚78.94748718160037$ disk./.usageGB‚1176.374610900879 disk.sda2.in‚0 disk.sda2.out‚0# disk.dm-0.out‚97321.66845703125 disk.sda.out‚ 149.37890625 disk.sda3.out‚ 149.37890625! proc.memory.rssMB‚ 2836.296875 disk.dm-0.in‚12261.62109375$ disk.nvme0c0n1.in‚12261.62109375" disk.nvme0n1.in‚12261.62109375& disk.nvme0n1.out‚97321.66845703125 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.7109375* proc.memory.percent‚17.311382293701172 cpu‚25.35784357325199 network.sent‚ 7.751734e+06 network.recv‚ 1.696441e+06 disk.sda1.in‚0XÁøw‘›‚(:† ¬ûêÔª¯Ôù gpu.0.gpu‚22.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚102.782' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.5564 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ñŽ{œ‚(:ð ¬ûêÔ—ÂÞü network.recv‚ 1.700961e+06( disk./.usagePercent‚78.9516954334482 disk.dm-1.out‚ 149.796875 disk.drbd1260.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 149.796875$ memory_percent‚98.38314056396484% proc.memory.availableMB‚ 264.90625% disk./.usageGB‚1176.4373168945312 disk.dm-0.in‚12261.62109375 disk.sda.in‚0" disk.nvme0n1.in‚12261.62109375! proc.memory.rssMB‚ 2836.296875 disk.sda1.out‚0 disk.drbd1260.out‚ 0.71484375 disk.sda2.in‚0 disk.sda3.in‚0& disk.nvme0n1.out‚97322.15966796875* proc.memory.percent‚17.311382293701172 proc.cpu.threads‚17 network.sent‚ 7.757969e+06 disk.sda1.in‚0 disk.dm-1.in‚0# disk.dm-0.out‚97322.15966796875 disk.sda.out‚ 149.796875$ disk.nvme0c0n1.in‚12261.62109375( disk.nvme0c0n1.out‚97322.15966796875 cpu‚25.357639269602522ºŸf‚(:… »ûêÔ÷ gpu.0.gpu‚22.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚104.425' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.885 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0´ê~vž‚(:ë »ûêÔŸ—Óø$ memory_percent‚98.38478565216064* proc.memory.percent‚17.311382293701172 network.sent‚ 7.762766e+06 disk.sda.in‚0 disk.sda.out‚150.125 disk.sda1.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17 network.recv‚ 1.703711e+06 disk.sda2.out‚0 disk.sda3.in‚0$ disk.nvme0c0n1.in‚12261.62109375( disk.nvme0c0n1.out‚97643.50146484375 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.71484375 disk.sda2.in‚0 disk.dm-0.in‚12261.62109375# disk.dm-0.out‚97643.50146484375 disk.dm-1.out‚150.125( proc.memory.availableMB‚ 264.63671875! proc.memory.rssMB‚ 2836.296875 cpu‚25.357375140459187) disk./.usagePercent‚78.95426574909197% disk./.usageGB‚1176.4756164550781 disk.sda3.out‚150.125" disk.nvme0n1.in‚12261.62109375& disk.nvme0n1.out‚97643.50146484375™(%2Ÿ- _wandbruntime‚5258  _runtime‚5258ñœuÅ ¿ _runtime‚ 5258.87096155 _step‚250 & lr-AdamW/pg1‚0.0008521572029333148 & lr-AdamW/pg2‚0.0008521572029333148 trainer/global_step‚3099 $ _timestamp‚1.788526020651457e+09&ߢlÊ¡Ä & lr-AdamW/pg1‚0.0008521572029333148 & lr-AdamW/pg2‚0.0008521572029333148 trainer/global_step‚3099 $ _timestamp‚1.788526020651457e+09 _runtime‚ 5258.87096155 _step‚250úÝÕ)2¢- _wandbruntime‚5258  _runtime‚52585Û !Ü'£ÖO eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚133063011503616 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6148275462144 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6254674398208 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚664 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22253401 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚58603121 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚125592723435520 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚251 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚131741016900608 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚84067124 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚58603123 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6148041794048 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34966852 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚139317685901824 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚133062987563520 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 _runtime‚5258.872287729 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚125592741438464 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚125592741438464 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 142670245 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6148041794048 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚84067124 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚119578229849232 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34966850 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚3099 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚58603121 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚58603123 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚131740765229568 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 142696590 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚125592723435520½Ñä´á'¤ÛO PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚84093467 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚131740765229568 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6130620324455 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6130849305535 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚58603121 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚125592723435520 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚251 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚131741016900608 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚84067124 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚58603123 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6148041794048 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34966852 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚139317685901824 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚133062987563520 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 _runtime‚5258.872287729 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚125592741438464 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚125592741438464 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 142670245 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6148041794048 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚84067124 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚119578229849232 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34966850 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚3099 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚58603121 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚58603123 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚131740765229568 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 142696590 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚125592723435520 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚133063011503616 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6148275462144 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6254674398208 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚664 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22253401 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚84120185 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚139361367159296 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚133104695473664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 $ _timestamp‚1.788526021659056e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚58621396 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚661 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚125632027475456 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚ 5259.879784 _step‚252 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6150239501312 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚58621398 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 280120832 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚84120185 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6132807794729 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 142741583 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚84093816 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 142715212 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57237623 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 304060928 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚139361063098368 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22259971 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚131782266976768 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚125632009472512 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6132578813441 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚84093816 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 S?DeviceStatsMonitor.on_train_batch_end/requested_b‘$u ytes.all.freed‚125748321980737 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚57238286 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚125632009472512 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 142741583 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 142715212 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚125632027475456 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6150005819904 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6256391564800 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚119615760206656 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚119615743167296 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34977652 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚ 5259.879784 _step‚252 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6150239501312 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚58621398 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 280120832 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚84120185 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6132807794729 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 142741583 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚84093816 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 142715212 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57237623 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 304060928 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚139361063098368 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22259971 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚131782266976768 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚125632009472512 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6132578813441 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚84093816 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚125748321980737 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚57238286 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚125632009472512 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 142741583 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 142715212 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚125632027475456 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6150005819904 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6256391564800 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚119615760206656 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚119615743167296 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34977652 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚84120185 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚139361367159296 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚133104695473664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 $ _timestamp‚1.788526021659056e+09üt®“_2¨- _wandbruntime‚5259  _runtime‚5259m¿,\©Ö %loss/total_loss‚44.81576876011572 trainer/global_step‚3099 .loss/FRt1/detection_loss‚2.752567760872118 /loss/FRt2/assignment_loss‚3.204901178677877 1loss/SRqqt1/assignment_loss‚1.557186558842659 2loss/SRqqt2/assignment_loss‚2.5146212689578533 1loss/FBt3/assignment_loss‚0.13725026356987655 1loss/SRqqt4/assignment_loss‚2.057937374959389 /loss/FRt3/detection_loss‚1.4565014725923537 1loss/SRqqt2/detection_loss‚2.5983505185161317 /loss/FBt2/detection_loss‚2.7606812665859857 .loss/FRt2/detection_loss‚3.210782364010811 0loss/FRt4/assignment_loss‚3.1507295682316734 /loss/FBt3/detection_loss‚2.3278329968452454 2loss/SRqqt3/assignment_loss‚1.1459625871763344 1loss/SRqqt1/detection_loss‚1.8374833607425292 1loss/SRqqt3/detection_loss‚1.0279998676407904 epoch‚2 % _timestamp‚1.7885260216622643e+09 1loss/FBt2/assignment_loss‚0.17728522268498637 1loss/FBt4/assignment_loss‚0.12436629465921531 1loss/SRqqt4/detection_loss‚2.9621876180171967 /loss/FBt1/detection_loss‚2.4840010946447197 _runtime‚5259.882120333 /loss/FRt3/assignment_loss‚1.415964376926422 /loss/FRt4/detection_loss‚1.6777645819303062 /loss/FBt4/detection_loss‚0.7656145616378071 /loss/FRt1/assignment_loss‚3.198062853379683 _step‚253 1loss/FBt1/assignment_loss‚0.26973374801375694DµLÜaªÛ 1loss/SRqqt3/detection_loss‚1.0279998676407904 epoch‚2 % _timestamp‚1.7885260216622643e+09 1loss/FBt2/assignment_loss‚0.17728522268498637 1loss/FBt4/assignment_loss‚0.12436629465921531 1loss/SRqqt4/detection_loss‚2.9621876180171967 /loss/FBt1/detection_loss‚2.4840010946447197 _runtime‚5259.882120333 /loss/FRt3/assignment_loss‚1.415964376926422 /loss/FRt4/detection_loss‚1.6777645819303062 /loss/FBt4/detection_loss‚0.7656145616378071 /loss/FRt1/assignment_loss‚3.198062853379683 _step‚253 1loss/FBt1/assignment_loss‚0.26973374801375694 %loss/total_loss‚44.81576876011572 trainer/global_step‚3099 .loss/FRt1/detection_loss‚2.752567760872118 /loss/FRt2/assignment_loss‚3.204901178677877 1loss/SRqqt1/assignment_loss‚1.557186558842659 2loss/SRqqt2/assignment_loss‚2.5146212689578533 1loss/FBt3/assignment_loss‚0.13725026356987655 1loss/SRqqt4/assignment_loss‚2.057937374959389 /loss/FRt3/detection_loss‚1.4565014725923537 1loss/SRqqt2/detection_loss‚2.5983505185161317 /loss/FBt2/detection_loss‚2.7606812665859857 .loss/FRt2/detection_loss‚3.210782364010811 0loss/FRt4/assignment_loss‚3.1507295682316734 /loss/FBt3/detection_loss‚2.3278329968452454 2loss/SRqqt3/assignment_loss‚1.1459625871763344 1loss/SRqqt1/detection_loss‚1.8374833607425292ý«tY\›«‚(: ÊûêÔàˆ–÷ gpu.0.gpu‚7.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚100.905' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚20.180999999999997 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0$š/;…¬‚(:ú ÊûêԿËø disk.dm-1.in‚0 disk.dm-0.in‚12305.29296875 disk.sda.out‚ 150.57421875 disk.drbd1260.out‚ 0.72265625# disk.dm-0.out‚97904.67626953125 disk.sda.in‚0( disk.nvme0c0n1.out‚97904.67626953125$ memory_percent‚98.39084148406982* proc.memory.percent‚17.311382293701172 proc.cpu.threads‚17 disk.dm-1.out‚ 150.57421875 disk.drbd1260.in‚0 network.sent‚ 7.767224e+06) disk./.usagePercent‚78.96270375742053& disk.nvme0n1.out‚97904.67626953125 disk.sda1.in‚0 disk.sda3.in‚0$ disk.nvme0c0n1.in‚12305.29296875 disk.sda1.out‚0 disk.sda3.out‚ 150.57421875( proc.memory.availableMB‚ 263.64453125! proc.memory.rssMB‚ 2836.296875 cpu‚25.357171574261464 disk.sda2.in‚0 disk.sda2.out‚0 network.recv‚ 1.707162e+06% disk./.usageGB‚1176.6013488769531" disk.nvme0n1.in‚12305.29296875©z÷%›­‚(: ÙûêÔÇ‘ö gpu.0.gpu‚13.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚94.091' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚18.818199999999997 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%´dƒ®‚(:ø ÙûêÔ¼á¨ù! proc.memory.rssMB‚ 2836.296875 network.recv‚ 1.713443e+06) disk./.usagePercent‚78.96270452544314# disk.dm-0.out‚98394.09814453125 disk.sda3.out‚ 151.05859375$ disk.nvme0c0n1.in‚12413.31640625 disk.dm-1.in‚0 disk.sda2.out‚0* proc.memory.percent‚17.311382293701172$ disk./.usageGB‚1176.601360321045 disk.sda3.in‚0" disk.nvme0n1.in‚12413.31640625 disk.drbd1260.out‚ 0.72265625 disk.sda2.in‚0$ memory_percent‚98.39205741882324 cpu‚25.356904893917008 proc.cpu.threads‚17 network.sent‚ 7.875623e+06 disk.dm-0.in‚12413.31640625( disk.nvme0c0n1.out‚98394.09814453125 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda.out‚ 151.05859375& disk.nvme0n1.out‚98394.09814453125 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 151.05859375' proc.memory.availableMB‚ 263.4453125´‹Ù믂(:… èûêÔê¤ÿö gpu.0.gpu‚19.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚92.922' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.5844 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0>µ*2x°‚(:í èûêÔäÛø$ disk.nvme0c0n1.in‚12413.31640625 disk.sda1.in‚0! proc.memory.rssMB‚ 2836.296875 proc.cpu.threads‚17 network.sent‚ 7.881979e+06 disk.dm-0.in‚12413.31640625" disk.nvme0n1.in‚12413.31640625 disk.sda1.out‚0 disk.dm-1.in‚0* proc.memory.percent‚17.311382293701172( disk.nvme0c0n1.out‚98813.21728515625 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda.out‚ 151.34375& disk.nvme0n1.out‚98813.21728515625 disk.drbd1260.out‚ 0.7265625( proc.memory.availableMB‚ 263.69140625 cpu‚25.35656985874993 network.recv‚ 1.718175e+06) disk./.usagePercent‚78.98105719355776 disk.sda3.out‚ 151.34375 disk.dm-1.out‚ 151.34375 disk.drbd1260.in‚0# memory_percent‚98.3905553817749$ disk./.usageGB‚1176.874828338623# disk.dm-0.out‚98813.21728515625 disk.sda.in‚0 disk.sda3.in‚0wz[‘±‚(:† ÷ûêÔèö÷ gpu.0.gpu‚44.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚113.667' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.7334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ìæÝz²‚(:ï ÷ûêÔàϺø! proc.memory.rssMB‚ 2836.296875$ disk./.usageGB‚1176.874828338623 disk.sda2.out‚0 disk.dm-0.in‚12413.31640625# disk.dm-0.out‚98986.03369140625 disk.sda.in‚0$ disk.nvme0c0n1.in‚12413.31640625* proc.memory.percent‚17.311382293701172 proc.cpu.threads‚17 network.sent‚ 7.886958e+06( disk.nvme0c0n1.out‚98986.03369140625& disk.nvme0n1.out‚98986.03369140625 cpu‚25.3562216625794) disk./.usagePercent‚78.98105719355776 disk.dm-1.out‚ 151.7109375 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.7265625 disk.sda2.in‚0 disk.sda.out‚ 151.7109375 disk.sda3.in‚0 disk.sda3.out‚ 151.7109375 network.recv‚ 1.721425e+06 disk.dm-1.in‚0" disk.nvme0n1.in‚12413.31640625 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚98.38905334472656$ proc.memory.availableMB‚263.9375Õ{ÏS³‚(:„ †üêÔò¼Ë÷ gpu.0.gpu‚20.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚113.64' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.728 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0½>DÁƒ´‚(:ø †üêÔÙ™÷ù disk.sda3.out‚ 152.05859375( disk.nvme0c0n1.out‚99192.92431640625 disk.sda1.out‚0 disk.sda2.in‚0 proc.cpu.threads‚17 network.sent‚ 7.892156e+06 network.recv‚ 1.724964e+06$ disk./.usageGB‚1176.874828338623$ disk.nvme0c0n1.in‚12413.31640625 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.73046875 disk.sda2.out‚0 disk.sda.in‚0 disk.sda3.in‚0" disk.nvme0n1.in‚12413.31640625& disk.nvme0n1.out‚99192.92431640625 disk.sda1.in‚0 disk.drbd1260.in‚0 disk.dm-0.in‚12413.31640625# disk.dm-0.out‚99192.92431640625' proc.memory.availableMB‚ 263.9453125) disk./.usagePercent‚78.98105719355776 disk.dm-1.out‚ 152.05859375 disk.sda.out‚ 152.05859375$ memory_percent‚98.38900566101074! proc.memory.rssMB‚ 2836.296875* proc.memory.percent‚17.311382293701172 cpu‚25.355957835909116ëj{7›µ‚(: •üêÔóüª÷ gpu.0.gpu‚27.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚97.846' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚19.569200000000002 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¹ýŸ5‚¶‚(:÷ •üêÔƧ’ø disk.drbd1260.out‚ 0.73046875 disk.dm-0.in‚ 12539.2265625# disk.dm-0.out‚99510.01025390625 disk.sda.in‚0# disk.nvme0c0n1.in‚ 12539.2265625 disk.sda2.out‚0 disk.sda.out‚ 152.62890625( proc.memory.availableMB‚ 263.44921875! proc.memory.rssMB‚ 2836.296875* proc.memory.percent‚17.311382293701172 cpu‚25.355726470577366 network.sent‚ 7.898114e+06) disk./.usagePercent‚79.01474804107131% disk./.usageGB‚1177.3768463134766( disk.nvme0c0n1.out‚99510.01025390625 disk.sda1.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 152.62890625 disk.sda3.in‚0! disk.nvme0n1.in‚ 12539.2265625& disk.nvme0n1.out‚99510.01025390625 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.sda3.out‚ 152.62890625$ memory_percent‚98.39203357696533 proc.cpu.threads‚17 network.recv‚ 1.729238e+06”7êm2·- _wandbruntime‚5340  _runtime‚5340eá·KƸÀ trainer/global_step‚3149 $ _timestamp‚1.788526102114035e+09 _runtime‚5340.334958873 _step‚254 & lr-AdamW/pg1‚0.0008486740785829605 & lr-AdamW/pg2‚0.0008486740785829605\¦Fa˹Štrainer/global_step‚3149 $ _timestamp‚1.788526102114035e+09 _runtime‚5340.334958873 _step‚254 & lr-AdamW/pg1‚0.0008486740785829605 & lr-AdamW/pg2‚0.0008486740785829605þæ2º- _wandbruntime‚5340  _runtime‚5340ψÐÜ »ÖO ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚127557043288064 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚59516873 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35506952 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6246491168768 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 294814208 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚141501882890752 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚127557025285120 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚59516871 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 318754304 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚85405076 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6228558364963 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚133803534456832 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚133803282785792 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚135147186065920 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚127683535066475 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚127557025285120 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚85431419 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 trainer/global_step‚3149 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚127557043288064 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚58106404 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22600142 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚85431419 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35506950 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7885261021153834e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚133803282785792 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6246257500672 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚85405076 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22599454 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚59516871 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273« .&á'¼ÛO PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚59516871 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 318754304 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚85405076 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6228558364963 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚133803534456832 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚133803282785792 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚135147186065920 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚127683535066475 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚127557025285120 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚85431419 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 trainer/global_step‚3149 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚127557043288064 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚58106404 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22600142 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚85431419 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35506950 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7885261021153834e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚133803282785792 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6246257500672 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚85405076 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22599454 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚59516871 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚127557043288064 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚59516873 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35506952 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6246491168768 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 294814208 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚141501882890752 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚127557025285120 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608ÿ= ¬2½- _wandbruntime‚5341  _runtime‚5341ÌÁ¼æ&¾àM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 144993285 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚121492278077856 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚59535146 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58123940 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6248221526528 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚133844532848640 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 _step‚256 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚127596311322112 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚59535146 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚85431768 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚58124631 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚691 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 trainer/global_step‚3149 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 144993285 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚127596329325056 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _runtime‚5341.792673902 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚59535148 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22606877 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚127596311322112 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚133844784532992 % _timestamp‚1.7885261035669656e+09 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚141545267275776 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2_m•˜ë&¿åM I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6248221526528 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 144993285 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚121492278077856 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚59535146 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58123940 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6248221526528 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚133844532848640 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 _step‚256 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚127596311322112 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚59535146 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚85431768 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚58124631 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚691 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 trainer/global_step‚3149 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 144993285 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚127596329325056 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _runtime‚5341.792673902 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚59535148 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22606877 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚127596311322112 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚133844784532992 % _timestamp‚1.7885261035669656e+09 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚141545267275776 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚60430623 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36047050 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6344473207296 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚129521345137664 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚135865800342016 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 trainer/global_step‚3199 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22946529 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚129521327134720 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 # _timestamp‚1.78852618836599e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚60430621 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22947226 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6326725386551 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚60430623 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58994278 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚129657744957743 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6344706875392 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚60430621 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚135866052013056 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚129521345137664 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚137231408508416 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚123331265591632 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚135865800342016 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚86769371 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚5426.586084163 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 147173649 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6454610422784 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6344706875392 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6454321900032 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚699 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36047052 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712£D$bß'ÒÙO OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚58993579 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚137231384568320 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚86743028 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚697 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚143685706468352 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚129521327134720 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6344473207296 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚143686018931200 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462848 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36047052 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 147199994 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚129657990978183 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚86743028 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚123331248552272 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚60430623 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36047050 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6344473207296 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚129521345137664 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚135865800342016 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 trainer/global_step‚3199 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22946529 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚129521327134720 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 # _timestamp‚1.78852618836599e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚60430621 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22947226 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6326725386551 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚60430623 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58994278 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚129657744957743 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6344706875392 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚60430621 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚135866052013056 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚129521345137664 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚137231408508416 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚123331265591632 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚135865800342016 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚86769371 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚5426.586084163 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 147173649 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6454610422784 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6344706875392 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6454321900032 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚699 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚260 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚129560631174656 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚675 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.cu?Î fcrrent‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚129560613171712 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6328683875745 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚60448898 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚60448898 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚60448896 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22954510 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6456309431808 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚86769720 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 147218616 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚123368795949056 % _timestamp‚1.7885261901020143e+09 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚137273092478464 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 147244987 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59011689 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 296898048 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚60448896 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 147244987 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6346437233152 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚143729377970176 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚135907302089216 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚129560631174656 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36057852 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚673 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6456606329856 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚59012364 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚129560613171712 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6346670914560 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚123368778909696 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 320838144 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36057852 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚673 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6456606329856 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚59012364 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚129560613171712 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6346670914560 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚123368778909696 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 320838144 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚260 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚129560631174656 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚675 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚129560613171712 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6328683875745 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚60448898 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚60448898 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚60448896 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22954510 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6456309431808 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚86769720 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 147218616 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚123368795949056 % _timestamp‚1.7885261901020143e+09 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚137273092478464 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 147244987 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59011689 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 296898048 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚60448896 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 147244987 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6346437233152 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚143729377970176„ !T2Ö- _wandbruntime‚5428  _runtime‚5428bÞÌ]×× 0loss/FRt2/assignment_loss‚4.2218445268544285 1loss/FBt4/assignment_loss‚0.18233157879987652 /loss/FRt2/detection_loss‚3.2633291455832394 /loss/FRt4/detection_loss‚1.5435695345668743 /loss/FBt4/detection_loss‚0.8435191373592095 % _timestamp‚1.7885261901046293e+09 /loss/FRt1/assignment_loss‚4.041719886389646 0loss/FRt3/assignment_loss‚1.4980652800628118 1loss/SRqqt3/assignment_loss‚1.045851086459418 0loss/SRqqt1/detection_loss‚2.185703073823175 1loss/SRqqt4/detection_loss‚2.7484978670659275 /loss/FBt2/detection_loss‚2.2374503752764534 /loss/FBt3/detection_loss‚2.8644511997699738 2loss/SRqqt4/assignment_loss‚2.4930146148671275 1loss/FBt1/assignment_loss‚0.35114439722670604 1loss/SRqqt2/detection_loss‚2.5245227909088133 /loss/FBt1/detection_loss‚2.0971272336809257 trainer/global_step‚3199 2loss/SRqqt1/assignment_loss‚1.6338956864594028 1loss/FBt3/assignment_loss‚0.16575538174947724 /loss/FRt3/detection_loss‚1.4854145125619003 1loss/FBt2/assignment_loss‚0.34235667827370514 0loss/SRqqt3/detection_loss‚0.896956791934686 %loss/total_loss‚46.75668568730262 _step‚261 epoch‚2 /loss/FRt4/assignment_loss‚2.870215746760368 2loss/SRqqt2/assignment_loss‚1.9993771010637282 .loss/FRt1/detection_loss‚3.220572059804743 _runtime‚5428.324207811ÿf˜bØÜ 1loss/FBt2/assignment_loss‚0.34235667827370514 0loss/SRqqt3/detection_loss‚0.896956791934686 %loss/total_loss‚46.75668568730262 _step‚261 epoch‚2 /loss/FRt4/assignment_loss‚2.870215746760368 2loss/SRqqt2/assignment_loss‚1.9993771010637282 .loss/FRt1/detection_loss‚3.220572059804743 _runtime‚5428.324207811 0loss/FRt2/assignment_loss‚4.2218445268544285 1loss/FBt4/assignment_loss‚0.18233157879987652 /loss/FRt2/detection_loss‚3.2633291455832394 /loss/FRt4/detection_loss‚1.5435695345668743 /loss/FBt4/detection_loss‚0.8435191373592095 % _timestamp‚1.7885261901046293e+09 /loss/FRt1/assignment_loss‚4.041719886389646 0loss/FRt3/assignment_loss‚1.4980652800628118 1loss/SRqqt3/assignment_loss‚1.045851086459418 0loss/SRqqt1/detection_loss‚2.185703073823175 1loss/SRqqt4/detection_loss‚2.7484978670659275 /loss/FBt2/detection_loss‚2.2374503752764534 /loss/FBt3/detection_loss‚2.8644511997699738 2loss/SRqqt4/assignment_loss‚2.4930146148671275 1loss/FBt1/assignment_loss‚0.35114439722670604 1loss/SRqqt2/detection_loss‚2.5245227909088133 /loss/FBt1/detection_loss‚2.0971272336809257 trainer/global_step‚3199 2loss/SRqqt1/assignment_loss‚1.6338956864594028 1loss/FBt3/assignment_loss‚0.16575538174947724 /loss/FRt3/detection_loss‚1.4854145125619003…³™XƒÙ‚(:‚ ïüêÔ®Ú¾÷ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚106.95' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.39 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0*‰$uÚ‚(:ê ïüêÔ¤Åü÷ disk.sda.in‚0 disk.sda.out‚ 155.171875 disk.sda3.in‚0( disk.nvme0c0n1.out‚101568.6103515625& disk.nvme0n1.out‚101568.6103515625 network.sent‚ 8.034421e+06 disk.dm-1.out‚ 155.171875 disk.sda2.out‚0 disk.dm-0.in‚ 12722.59375# disk.dm-0.out‚101568.6103515625$ memory_percent‚98.38511943817139( proc.memory.availableMB‚ 264.58203125 cpu‚25.35426582237724 network.recv‚ 1.756238e+06 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda3.out‚ 155.171875! disk.nvme0c0n1.in‚ 12722.59375 disk.nvme0n1.in‚ 12722.59375 proc.cpu.threads‚17% disk./.usageGB‚1177.2363319396973 disk.dm-1.in‚0! proc.memory.rssMB‚ 2836.296875* proc.memory.percent‚17.311382293701172) disk./.usagePercent‚79.00531800354756 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.74609375ü|¹(‘Û‚(:† þüêÔѸ‡ù gpu.0.gpu‚19.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚102.662' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.5324 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Í tÜ‚(:é þüêÔýŽ¹þ disk.sda1.in‚0 disk.drbd1260.out‚0.75 network.recv‚ 1.7619e+06 disk.nvme0n1.in‚ 12722.59375& disk.nvme0n1.out‚102058.3525390625# disk.dm-0.out‚102058.3525390625 disk.sda3.in‚0 proc.cpu.threads‚17 disk.drbd1260.in‚0 disk.sda2.out‚0 disk.sda.in‚0 disk.sda.out‚ 155.49609375 disk.sda3.out‚ 155.49609375$ memory_percent‚98.38664531707764( proc.memory.availableMB‚ 264.33203125 cpu‚25.354010787980084 disk.dm-1.in‚0 network.sent‚ 8.141929e+06) disk./.usagePercent‚79.00689296189822% disk./.usageGB‚1177.2597999572754! disk.nvme0c0n1.in‚ 12722.59375( disk.nvme0c0n1.out‚102058.3525390625 disk.sda2.in‚0 disk.dm-0.in‚ 12722.59375 disk.sda1.out‚0 disk.dm-1.out‚ 155.49609375! proc.memory.rssMB‚ 2836.296875* proc.memory.percent‚17.3113822937011726£xœÝ‚(:‘ ýêÔ‡ÇØö gpu.0.gpu‚10.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚104.585' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚20.916999999999998 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0üÁV¡uÞ‚(:ê ýêÔÉöè÷% disk./.usageGB‚1177.2597999572754" disk.nvme0c0n1.in‚ 12897.984375 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 156.01953125 cpu‚25.353881127767313& disk.nvme0n1.out‚102308.7666015625 disk.dm-1.in‚0 disk.dm-1.out‚ 156.01953125# disk.dm-0.out‚102308.7666015625 disk.sda.in‚0 disk.sda.out‚ 156.01953125! proc.memory.rssMB‚ 2836.296875 network.sent‚ 8.147772e+06 network.recv‚ 1.766377e+06) disk./.usagePercent‚79.00689296189822 disk.drbd1260.out‚0.75 disk.sda2.out‚0$ memory_percent‚98.38676452636719$ proc.memory.availableMB‚264.3125* proc.memory.percent‚17.311382293701172( disk.nvme0c0n1.out‚102308.7666015625 disk.nvme0n1.in‚ 12897.984375 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.dm-0.in‚ 12897.984375 proc.cpu.threads‚17VÈû œß‚(:‘ œýêÔĵßù gpu.0.gpu‚19.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚103.766' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚20.753200000000003 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÎÆþ{à‚(:ð œýêÔ÷é©ü% disk./.usageGB‚1177.2597999572754" disk.nvme0c0n1.in‚ 12897.984375 disk.dm-1.out‚ 156.5078125 disk.drbd1260.out‚ 0.75390625 disk.sda2.in‚0 disk.sda3.in‚0 cpu‚25.35369222915675 network.sent‚ 8.154062e+06 network.recv‚ 1.771062e+06' disk.nvme0n1.out‚102563.87158203125 disk.sda1.in‚0 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda.out‚ 156.5078125& proc.memory.availableMB‚ 264.578125) disk.nvme0c0n1.out‚102563.87158203125 disk.nvme0n1.in‚ 12897.984375 disk.sda2.out‚0 disk.sda3.out‚ 156.5078125# memory_percent‚98.3851432800293! proc.memory.rssMB‚ 2836.296875 proc.cpu.threads‚17 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.dm-0.in‚ 12897.984375$ disk.dm-0.out‚102563.87158203125* proc.memory.percent‚17.311382293701172) disk./.usagePercent‚79.00689296189822‘<|?á‚(:ô «ýêÔ¹Óú÷ disk.sda.in‚0 disk.sda3.in‚0 network.recv‚ 1.774562e+06" disk.nvme0c0n1.in‚ 12897.984375 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.sda.out‚ 156.88671875# memory_percent‚98.3851432800293& proc.memory.availableMB‚ 264.578125) disk./.usagePercent‚79.00689296189822 disk.sda1.in‚0 disk.dm-1.out‚ 156.88671875 disk.sda3.out‚ 156.88671875! proc.memory.rssMB‚ 2836.296875* proc.memory.percent‚17.311382293701172 cpu‚25.353452824015424 proc.cpu.threads‚17 network.sent‚ 8.159307e+06% disk./.usageGB‚1177.2597999572754 disk.nvme0n1.in‚ 12897.984375 disk.sda1.out‚0 disk.dm-0.in‚ 12897.984375$ disk.dm-0.out‚102986.76220703125) disk.nvme0c0n1.out‚102986.76220703125' disk.nvme0n1.out‚102986.76220703125 disk.drbd1260.out‚ 0.75390625 disk.sda2.out‚0¬7Bœâ‚(:‘ «ýêÔ˨™ù gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚137.364' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.472800000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡Ò3š|ã‚(:ñ ºýêÔ ÎÏø disk.nvme0n1.in‚ 12897.984375 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.dm-0.in‚ 12897.984375* proc.memory.percent‚17.311382293701172 cpu‚25.3532123022686 proc.cpu.threads‚17" disk.nvme0c0n1.in‚ 12897.984375) disk.nvme0c0n1.out‚103304.09326171875' disk.nvme0n1.out‚103304.09326171875 disk.sda2.out‚0$ disk.dm-0.out‚103304.09326171875 disk.sda.in‚0 disk.sda3.out‚ 157.18359375# memory_percent‚98.3851432800293 network.sent‚ 8.164308e+06 network.recv‚ 1.778165e+06) disk./.usagePercent‚79.00689296189822 disk.sda1.in‚0 disk.sda3.in‚0& proc.memory.availableMB‚ 264.578125! proc.memory.rssMB‚ 2836.296875% disk./.usageGB‚1177.2597999572754 disk.sda1.out‚0 disk.dm-1.out‚ 157.18359375 disk.drbd1260.out‚ 0.7578125 disk.sda.out‚ 157.183593752Œ¬ò‘ä‚(:† ºýêÔ²¯òù gpu.0.gpu‚38.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚112.053' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.4106 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0dzq¾2å- _wandbruntime‚5509  _runtime‚5509X¸ö£ÇæÁ & lr-AdamW/pg1‚0.0008414652256133419 & lr-AdamW/pg2‚0.0008414652256133419 trainer/global_step‚3249 % _timestamp‚1.7885262711100397e+09 _runtime‚5509.329471469 _step‚262ÞÙ]ÌçÆ _step‚262 & lr-AdamW/pg1‚0.0008414652256133419 & lr-AdamW/pg2‚0.0008414652256133419 trainer/global_step‚3249 % _timestamp‚1.7885262711100397e+09 _runtime‚5509.329471469†æŽ=À2è- _wandbruntime‚5509  _runtime‚5509XP³=ÙéÖO ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6442688913920 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚686 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 % _timestamp‚1.7885262711109989e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚145870119074816 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚137928569569280 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 149425351 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚137928317898240 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚131632446889891 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚88107323 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23294076 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚139315607010816 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6424434445979 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚131485628984320 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36587152 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268544 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚137928569569280 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23294760 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6442688913920 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚131485628984320 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6442922582016 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 D:DeviceStatsMonitor.on_train_batch_start/active.al+‡üol.current‚26345 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚88080980 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚125207783462832 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6424663427059 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚88080980 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 trainer/global_step‚3249 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚139315583070720 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284328448 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _step‚263 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚131485646987264 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚137928317898240 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚131485646987264 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚684 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6442922582016 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚145869810806272 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚131632200869451 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚5509.330750087 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚88080980 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚125207783462832 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6424663427059 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚88080980 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 trainer/global_step‚3249 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚139315583070720 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284328448 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _step‚263 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚131485646987264 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚137928317898240 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚131485646987264 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚684 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6442922582016 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚145869810806272 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚131632200869451 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚5509.330750087 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚137969819645440 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 149470318 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6444652939776 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6444652939776 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 149496689 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚61362648 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚88107672 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚125245296780896 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 _step‚264 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286412288 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚131671935736509 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6426621916253 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚131524915021312 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚131671689715861 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚139357267040768 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚145913799712768 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚685 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36597954 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚61362646 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36597952 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 % _timestamp‚1.7885262720508103e+09 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 149496689 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23300791 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 149470318 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚61362648 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59898743 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310352384 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚137969567961088 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚59899428 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2:TŒáíåM ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚131524933024256 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚88134041 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚131671935736509 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6426621916253 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚131524915021312 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚131671689715861 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚139357267040768 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚145913799712768 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚685 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36597954 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚61362646 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36597952 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 % _timestamp‚1.7885262720508103e+09 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 149496689 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23300791 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 149470318 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚61362648 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59898743 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310352384 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚137969567961088 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚59899428 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚683 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚137969819645440 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 149470318 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6444652939776 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6444652939776 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 149496689 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚61362648 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚88107672 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚125245296780896 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 _step‚264 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286412288 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚127084284294672 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚710 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6654062622720 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6540904620544 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚133449930833920 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚148053844195840 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚133606902801599 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚133449948836864 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚133606656781159 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚127084301334032 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 % _timestamp‚1.7885263318629036e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚62258123 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚267 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚89445275 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23638064 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 151703398 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚139990835454464 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6540904620544 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6522372486487 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚141399805513216 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453¢íþá'þÛO YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚139991087125504 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6522601467567 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚89418932 trainer/global_step‚3299 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚127084284294672 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚710 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6654062622720 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6540904620544 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚133449930833920 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚148053844195840 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚133606902801599 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚133449948836864 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚133606656781159 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚127084301334032 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 % _timestamp‚1.7885263318629036e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚62258123 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚267 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚89445275 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23638064 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 151703398 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚139990835454464 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6540904620544 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6522372486487 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚141399805513216 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚89418932 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚62258121 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚712 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚139990835454464 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚62258121 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6541138288640 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚60765316 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 290619904 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309‹ÁîÁ2ÿ- _wandbruntime‚5572  _runtime‚5572B~Üe[€àM E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚62276396 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚62276396 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6542868646400 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚37138054 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 312449536 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6656342137344 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6524559956761 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚682 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6542868646400 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚133489234873856 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚148097519171072 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚62276398 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚89445624 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚37138052 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚141441489483264 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚89445624 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 151722020 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6656053627904 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚133489216870912 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 151748391 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚140032337201664 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚140032337201664 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚680 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _step‚268 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚140032085517312 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚133489234873856 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6524330975473 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 % _timestamp‚1.7885263340213082e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 151722020 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6543102327808 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚133646145627569 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚141441465543168 _runtime‚5572.252270885 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6543102327808 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60782010 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚62276396 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚62276396 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6542868646400 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚37138054 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 312449536 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6656342137344 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6524559956761 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚682 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6542868646400 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚133489234873856 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚148097519171072 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚62276398 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚89445624 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚37138052 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚141441489483264 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚89445624 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 151722020 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6656053627904 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚133489216870912 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 151748391 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚140032337201664 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚140032337201664 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚680 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _step‚268 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚140032085517312 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚133489234873856 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6524330975473 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 % _timestamp‚1.7885263340213082e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 151722020 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6543102327808 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚133646145627569 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚141441465543168 _runtime‚5572.252270885 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6543102327808 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60782010 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 PDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚37667350 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23983251 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚142053353010688 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚61650601 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚150238327310336 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6639353995264 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚37667352 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23983957 _step‚271 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚142053604681728 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚142053353010688 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6639120327168 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚90756884 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6620310526995 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7885264025279818e+09 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 153955100 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 316657152 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚135414232683520 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚128960802165872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6639353995264 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚135414232683520 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚128960802165872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6639353995264 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚37667350 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23983251 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚142053353010688 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚61650601 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚150238327310336 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608wÍ2•- _wandbruntime‚5643  _runtime‚5643ùÛÄ“å&–ßM XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6641084353024 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23990331 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6641318034432 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚63190148 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚680 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _step‚272 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚90783576 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 153973722 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚37678154 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚142094603073536 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚678 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚143525687985664 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚143525664045568 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6756028779520 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚128998349562656 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚150281692825088 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚63190148 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 154000093 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚135453518720512 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚142094854757888 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 trainer/global_step‚3349 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6622497997269 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 $ _timestamp‚1.788526404796677e+09 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 153973722 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚63190146 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚128998332523296 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 154000093 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚63190146 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6641318034432 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚135620601539277 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61668483 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚5643.021130835 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚90783576 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309ÖiJ»—äM T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚142094854757888 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 trainer/global_step‚3349 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6622497997269 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 $ _timestamp‚1.788526404796677e+09 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 153973722 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚63190146 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚128998332523296 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 154000093 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚63190146 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6641318034432 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚135620601539277 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61668483 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚5643.021130835 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚90783576 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6641084353024 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23990331 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6641318034432D#(/ VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚63190148 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚680 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _step‚272 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚90783576 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 153973722 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚37678154 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚142094603073536 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚678 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚143525687985664 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚143525664045568 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6756028779520 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚128998349562656 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚150281692825088 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚63190148 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 154000093 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚135453518720512 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608õ9Y2˜- _wandbruntime‚5643  _runtime‚5643Ù." [™Õ 0loss/FRt3/assignment_loss‚1.3700501854355271 1loss/FBt3/assignment_loss‚0.07080825727548885 1loss/SRqqt3/detection_loss‚0.8526347548663616 1loss/SRqqt4/detection_loss‚2.6526797874407335 /loss/FBt3/detection_loss‚1.8200342763553967 /loss/FRt2/detection_loss‚3.0220439434051514 0loss/FRt2/assignment_loss‚3.6456007682360134 1loss/FBt1/assignment_loss‚0.17344923584602867 epoch‚2 0loss/FRt4/assignment_loss‚3.1688036993145943 0loss/SRqqt2/detection_loss‚2.647969439111907 .loss/FBt1/detection_loss‚2.662811353802681 .loss/FBt4/detection_loss‚0.771574894558977 % _timestamp‚1.7885264048046086e+09 _step‚273 2loss/SRqqt1/assignment_loss‚1.4281332857078977 1loss/FBt2/assignment_loss‚0.16582756281941322 /loss/FRt3/detection_loss‚1.5729256378033676 trainer/global_step‚3349 #loss/total_loss‚43.443127573073 2loss/SRqqt4/assignment_loss‚1.9475035680965944 0loss/FBt4/assignment_loss‚0.1311994293970222 0loss/FRt1/assignment_loss‚3.1551097416877747 2loss/SRqqt2/assignment_loss‚1.5487389003472596 0loss/SRqqt1/detection_loss‚2.297751764456431 _runtime‚5643.025127768 2loss/SRqqt3/assignment_loss‚0.9564008359014988 /loss/FRt1/detection_loss‚3.2109950041770934 /loss/FRt4/detection_loss‚1.8342374509499808 .loss/FBt2/detection_loss‚2.335843796079809·(‘Â`šÚ 2loss/SRqqt4/assignment_loss‚1.9475035680965944 0loss/FBt4/assignment_loss‚0.1311994293970222 0loss/FRt1/assignment_loss‚3.1551097416877747 2loss/SRqqt2/assignment_loss‚1.5487389003472596 0loss/SRqqt1/detection_loss‚2.297751764456431 _runtime‚5643.025127768 2loss/SRqqt3/assignment_loss‚0.9564008359014988 /loss/FRt1/detection_loss‚3.2109950041770934 /loss/FRt4/detection_loss‚1.8342374509499808 .loss/FBt2/detection_loss‚2.335843796079809 0loss/FRt3/assignment_loss‚1.3700501854355271 1loss/FBt3/assignment_loss‚0.07080825727548885 1loss/SRqqt3/detection_loss‚0.8526347548663616 1loss/SRqqt4/detection_loss‚2.6526797874407335 /loss/FBt3/detection_loss‚1.8200342763553967 /loss/FRt2/detection_loss‚3.0220439434051514 0loss/FRt2/assignment_loss‚3.6456007682360134 1loss/FBt1/assignment_loss‚0.17344923584602867 epoch‚2 0loss/FRt4/assignment_loss‚3.1688036993145943 0loss/SRqqt2/detection_loss‚2.647969439111907 .loss/FBt1/detection_loss‚2.662811353802681 .loss/FBt4/detection_loss‚0.771574894558977 % _timestamp‚1.7885264048046086e+09 _step‚273 2loss/SRqqt1/assignment_loss‚1.4281332857078977 1loss/FBt2/assignment_loss‚0.16582756281941322 /loss/FRt3/detection_loss‚1.5729256378033676 trainer/global_step‚3349 #loss/total_loss‚43.443127573073‘òlg›‚(:… ÐþêÔꪄø gpu.0.gpu‚22.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚97.961' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.5922 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0* š}ˆœ‚(:ý ÐþêÔ­®—ø disk.sda1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.77734375 disk.sda2.out‚0 disk.dm-0.in‚13329.44140625! proc.memory.rssMB‚ 2836.296875$ disk.nvme0c0n1.in‚13329.44140625 disk.sda1.out‚0 disk.dm-1.in‚0$ disk.dm-0.out‚105879.82861328125 disk.sda.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 161.24609375* proc.memory.percent‚17.311382293701172 network.sent‚ 8.425049e+06% disk./.usageGB‚1176.2601165771484' disk.nvme0n1.out‚105879.82861328125 disk.sda.out‚ 161.24609375$ memory_percent‚98.38736057281494( proc.memory.availableMB‚ 264.21484375 cpu‚25.348575470195012 proc.cpu.threads‚17 network.recv‚ 1.823601e+06) disk./.usagePercent‚78.93980337146766) disk.nvme0c0n1.out‚105879.82861328125" disk.nvme0n1.in‚13329.44140625 disk.dm-1.out‚ 161.24609375 disk.sda2.in‚0R±p‘‚(:† ßþêÔÄäÿö gpu.0.gpu‚15.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚103.722' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.7444 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0§$û§ž‚(:ö ßþêÔôµî÷ disk.sda3.in‚0" disk.nvme0n1.in‚13329.44140625 disk.drbd1260.out‚ 0.77734375( proc.memory.availableMB‚ 264.21484375 cpu‚25.34827608964744 proc.cpu.threads‚17) disk./.usagePercent‚78.94399370278812 disk.sda.in‚0 disk.sda3.out‚ 161.671875$ disk.nvme0c0n1.in‚13329.44140625) disk.nvme0c0n1.out‚106220.94580078125 disk.drbd1260.in‚0 disk.sda2.out‚0$ disk.dm-0.out‚106220.94580078125 network.sent‚ 8.532386e+06' disk.nvme0n1.out‚106220.94580078125 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0 disk.sda2.in‚0 disk.dm-0.in‚13329.44140625$ memory_percent‚98.38736057281494% disk./.usageGB‚1176.3225555419922 disk.sda.out‚ 161.671875 disk.dm-1.out‚ 161.671875! proc.memory.rssMB‚ 2836.296875* proc.memory.percent‚17.311382293701172 network.recv‚ 1.828981e+06‘ú Ä‘Ÿ‚(:† îþêÔ¯‹¢ù gpu.0.gpu‚19.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚100.787' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.1574 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0iIüÐ} ‚(:ò îþêÔ—ÆÇú% disk.nvme0n1.out‚106788.384765625 disk.sda.out‚ 162.08984375$ memory_percent‚98.38738441467285! proc.memory.rssMB‚ 2836.296875 cpu‚25.347951043048962 proc.cpu.threads‚17) disk./.usagePercent‚78.94661317187847$ disk./.usageGB‚1176.361587524414 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚13329.44140625' proc.memory.availableMB‚ 264.2109375 network.recv‚ 1.832387e+06 disk.sda3.out‚ 162.08984375" disk.nvme0n1.in‚13329.44140625" disk.dm-0.out‚106788.384765625* proc.memory.percent‚17.311382293701172 disk.sda3.in‚0' disk.nvme0c0n1.out‚106788.384765625 disk.sda1.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 162.08984375 disk.drbd1260.in‚0 disk.drbd1260.out‚0.78125 disk.sda.in‚0 network.sent‚ 8.537606e+06$ disk.nvme0c0n1.in‚13329.44140625kbw‘¡‚(:† ýþêÔ¥‰´ö gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚112.742' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.5484 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Èò·|¢‚(:ñ ýþêÔ焘÷% disk./.usageGB‚1176.5100555419922 disk.sda1.out‚0 disk.drbd1260.in‚0" disk.dm-0.out‚107129.419921875( proc.memory.availableMB‚ 263.95703125! proc.memory.rssMB‚ 2836.296875 network.sent‚ 8.54357e+06 network.recv‚ 1.83685e+06 disk.sda3.out‚ 162.44921875' disk.nvme0c0n1.out‚107129.419921875" disk.nvme0n1.in‚13329.44140625% disk.nvme0n1.out‚107129.419921875$ memory_percent‚98.38893413543701 proc.cpu.threads‚17 disk.drbd1260.out‚0.78125 disk.sda2.in‚0 disk.dm-0.in‚13329.44140625* proc.memory.percent‚17.311382293701172 cpu‚25.347639266938554 disk.sda3.in‚0$ disk.nvme0c0n1.in‚13329.44140625 disk.sda1.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 162.44921875 disk.sda2.out‚0 disk.sda.in‚0 disk.sda.out‚ 162.44921875( disk./.usagePercent‚78.9565769851111ò{¦È‘£‚(:† ŒÿêÔûŸôø gpu.0.gpu‚28.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚110.733' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.1466 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚08àÍ*€¤‚(:õ ŒÿêÔÈ„îü network.sent‚ 8.550216e+06) disk./.usagePercent‚78.95657903317137% disk./.usageGB‚1176.5100860595703 disk.sda1.in‚0 disk.sda.in‚0! proc.memory.rssMB‚ 2836.296875* proc.memory.percent‚17.311382293701172 cpu‚25.347333640820796 disk.drbd1260.out‚ 0.78515625$ disk.nvme0c0n1.in‚13462.89453125 disk.dm-1.out‚ 163.1640625 disk.sda2.out‚0 disk.dm-0.in‚13462.89453125 disk.sda.out‚ 163.1640625# memory_percent‚98.3889102935791" disk.nvme0n1.in‚13462.89453125 disk.sda2.in‚0 disk.sda3.out‚ 163.1640625' proc.memory.availableMB‚ 263.9609375 proc.cpu.threads‚17 network.recv‚ 1.842231e+06( disk.nvme0c0n1.out‚107375.2822265625& disk.nvme0n1.out‚107375.2822265625 disk.sda1.out‚0 disk.dm-1.in‚0 disk.drbd1260.in‚0# disk.dm-0.out‚107375.2822265625 disk.sda3.in‚0ÖºN2¥- _wandbruntime‚5718  _runtime‚5718&ä*ǦÁ & lr-AdamW/pg1‚0.0008300568275115662 & lr-AdamW/pg2‚0.0008300568275115662 trainer/global_step‚3399 % _timestamp‚1.7885264800845306e+09 _runtime‚5718.304184267 _step‚274Ç«ÃôÌ§Æ & lr-AdamW/pg2‚0.0008300568275115662 trainer/global_step‚3399 % _timestamp‚1.7885264800845306e+09 _runtime‚5718.304184267 _step‚274 & lr-AdamW/pg1‚0.0008300568275115662’Tdö02¨- _wandbruntime‚5718  _runtime‚5718NχÛ'©ÕO BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚137378552536064 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚137378552536064 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6718248567503 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚137378534533120 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚64085621 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚64085623 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚38207450 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚137555568604575 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚144115870566912 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚62540638 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6853925079552 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚137555814625015 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292717056 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚62539931 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6854217796608 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚144116122237952 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 _step‚275 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚130837320037072 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚92094836 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24332481 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚64085623 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6737336033792 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 156206802 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38207452 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 trainer/global_step‚3399 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚152422420314624 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚5718.306507725 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚92121179 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚705 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6737569701888 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6737569701888 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚144116122237952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚92094836 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚137378534533120 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 316657152 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚144115870566912 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚92121179 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6718477548583 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6737336033792 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚130837337076432 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0*ÿ}É0!ªÚO YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚137378534533120 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚64085621 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚64085623 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚38207450 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚137555568604575 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚144115870566912 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚62540638 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6853925079552 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚137555814625015 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 292717056 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚62539931 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6854217796608 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚144116122237952 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 _step‚275 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚130837320037072 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚92094836 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24332481 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚64085623 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6737336033792 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 156206802 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38207452 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 trainer/global_step‚3399 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚152422420314624 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚5718.306507725 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚92121179 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚705 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6737569701888 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6737569701888 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚144116122237952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚92094836 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚137378534533120 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 316657152 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚144115870566912 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚92121179 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6718477548583 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6737336033792 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚130837337076432 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 L”Â0Ÿ°DDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚137378552536064 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚137378552536064 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6718248567503 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608“ðh2«- _wandbruntime‚5719  _runtime‚5719ß:æ&¬àM T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚144157372314112 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 _runtime‚5719.386419156 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 156251795 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚64103898 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚92121528 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 156251795 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚64103898 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚92147897 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6856224303616 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 trainer/global_step‚3399 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚64103896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 156225424 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚152466110791680 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚144157372314112 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚62558775 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217984 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚92121528 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚145609862547968 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚92147897 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚137417820570112 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62558099 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚64103896 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚38218254 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚144157120629760 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6720207056489 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6739533741056 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24339847 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6720436037777 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚130874867433856 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚676 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚137595303471633 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚674 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0+ðë&­åM D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 156225424 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚64103896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 156225424 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚152466110791680 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚144157372314112 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚62558775 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217984 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚92121528 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚145609862547968 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚92147897 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚137417820570112 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62558099 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚64103896 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚38218254 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚144157120629760 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6720207056489 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6739533741056 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24339847 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6720436037777 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚130874867433856 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚676 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚137595303471633 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚674 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚144157372314112 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 _runtime‚5719.386419156 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 156251795 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚64103898 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚92121528 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 156251795 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚64103898 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚92147897 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6856224303616 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 trainer/global_step‚3399 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚93459131 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6835785408512 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚132713854947632 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚63426741 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚146178639794176 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚64999371 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚146178388123136 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6835785408512 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6835551740416 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚64999373 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6954130907136 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚93432788 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚147652377080320 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284328448 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6953846578688 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚139342836382720 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚139342854385664 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚139530270536723 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 trainer/global_step‚3449 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 158432159 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚154606531927552 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚139342854385664 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚38747550 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24679191 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚132713837908272 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24679878 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268544 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚139530024516283 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚139342836382720 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚64999371 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚689 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚687 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚147652401020416 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚146178639794176 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚146178388123136 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6835785408512 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6835551740416 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚64999373 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6954130907136 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚93432788 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚147652377080320 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284328448 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6953846578688 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚139342836382720 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚139342854385664 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚139530270536723 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 trainer/global_step‚3449 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 158432159 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚154606531927552 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚139342854385664 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚38747550 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24679191 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚132713837908272 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24679878 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268544 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚139530024516283 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚139342836382720 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚64999371 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚689 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚687 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚147652401020416 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚146178639794176 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚93459131 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6835785408512 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚132713854947632 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚63426741 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚146178639794176 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚64999371 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63444259 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚146219889870336 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚146219638185984 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24685907 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _step‚280 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚139382122419712 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 % _timestamp‚1.7885265625893545e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 158477126 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6818145096997 trainer/global_step‚3449 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚65017646 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 158503497 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚65017648 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚154650217668096 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310352384 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚147694061050368 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚132751368265696 _runtime‚5800.814700192 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚154649907315712 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚132751385305056 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6956132677632 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286412288 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚139569513362693 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6837515766272 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6837749447680 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24686590 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚683 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 158503497 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6837515766272 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚139569759383341 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚93459480 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6818374078285 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚139382140422656 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 158477126 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚63444944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚139382140422656 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚685 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273tÔúh‡ ÃåM TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚65017646 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚93485849 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚93485849 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚139569759383341 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚93459480 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6818374078285 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚139382140422656 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 158477126 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚63444944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚139382140422656 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚685 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚93459480 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63444259 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚146219889870336 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚146219638185984 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24685907 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0´+Md C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _step‚280 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚139382122419712 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 % _timestamp‚1.7885265625893545e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 158477126 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6818145096997 trainer/global_step‚3449 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚65017646 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 158503497 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚65017648 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚154650217668096 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310352384 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚147694061050368 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚132751368265696 _runtime‚5800.814700192 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26369 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚154649907315712 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚132751385305056 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6956132677632 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286412288 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚139569513362693 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6837515766272 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6837749447680 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24686590 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚683 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 158503497 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6837515766272 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 % _timestamp‚1.7885266471455913e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6933767447040 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚134590355779472 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚65913123 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25024792 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 310365696 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 286425600 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚141307156235264 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚141307138232320 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7053974644224 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 _step‚283 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚3499 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚148240905679360 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚141307138232320 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7053688218624 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚715 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 160710206 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 160683861 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚134590372818832 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚64313159 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚64312442 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚65913121 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚94770740 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25025507 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚149736575582720 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚141504726448431 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚141504480427991 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6934001115136 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚94797083 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚148241157350400 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496G<êOÞØÛO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚94797083 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚717 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚39287652 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6914353629599 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6914124648519 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚141307156235264 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 % _timestamp‚1.7885266471455913e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6933767447040 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚134590355779472 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚65913123 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25024792 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 310365696 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 286425600 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚141307156235264 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚141307138232320 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7053974644224 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 _step‚283 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚3499 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚148240905679360 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚141307138232320 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7053688218624 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚715 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 160710206 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 160683861 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚134590372818832 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚64313159 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚64312442 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚65913121 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚94770740 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25025507 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚149736575582720 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚141504726448431 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚141504480427991 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6934001115136 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚94797083 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚148241157350400 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚149736599522816 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚148241157350400 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚156790574167040 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚148240905679360 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚156790263801344›ÚQ’2Ù- _wandbruntime‚5887  _runtime‚5887¹Ç:[å&ÚßM REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚65931398 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚156834251846144 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 160755199 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚141543969274401 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 160755199 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚149778283492864 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6916083137505 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 $ _timestamp‚1.788526648931203e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚148282407426560 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚141346424269312 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚3499 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64329929 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚134627886136896 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚65931398 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 160728828 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚699 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25032176 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6916312118793 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6935965154304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6935731472896 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚701 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 _step‚284 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6935731472896 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚94823801 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6935965154304 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217984 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚141544215295049 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 _runtime‚5887.156605468 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚39298454 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚149778259552768 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7055686135296 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚141346424269312 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚64330630 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚94797432 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚148282407426560 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚94797432 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚141346442272256 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚148282155742208 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚65931396 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚156833945688064 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7055968353280 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚39298452 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚134627903176256 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6935965154304 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217984 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚141544215295049 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 _runtime‚5887.156605468 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚39298454 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚149778259552768 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7055686135296 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚141346424269312 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚64330630 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚94797432 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚148282407426560 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚94797432 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚141346442272256 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚148282155742208 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚65931396 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚156833945688064 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7055968353280 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚39298452 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚134627903176256 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 160755199 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚141543969274401 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 160755199 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚149778283492864 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6916083137505 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 $ _timestamp‚1.788526648931203e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚148282407426560 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚141346424269312 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚3499 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64329929 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚134627886136896 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚65931398 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 160728828 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 339145216 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚699 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25032176 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020648 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6916312118793 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6935965154304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6935731472896 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚701 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 _step‚284 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6935731472896 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚94823801 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0œFÈÔM2Ü- _wandbruntime‚5887  _runtime‚5887üï¶^ÝØ epoch‚2 %loss/total_loss‚46.12060969501472 /loss/FRt2/detection_loss‚2.9099364907313614 0loss/FRt1/assignment_loss‚3.5316496726238364 2loss/SRqqt1/assignment_loss‚1.6177869649095968 1loss/SRqqt2/assignment_loss‚2.123323903977871 /loss/FRt1/detection_loss‚3.0054895904931156 /loss/FBt2/detection_loss‚2.5332283865321767 _runtime‚5887.160260937 0loss/FRt2/assignment_loss‚3.5722237672561254 2loss/SRqqt3/assignment_loss‚1.2153521278159363 0loss/FBt1/assignment_loss‚0.2762899082484864 1loss/SRqqt3/detection_loss‚1.1025548860709244 % _timestamp‚1.7885266489397187e+09 2loss/SRqqt4/assignment_loss‚1.8803740479052067 1loss/SRqqt1/detection_loss‚2.0851360214027492 /loss/SRqqt2/detection_loss‚2.80782699684302 0loss/SRqqt4/detection_loss‚3.310701002677282 0loss/FRt4/assignment_loss‚2.4913226536342075 1loss/FBt2/assignment_loss‚0.09638076117135018 1loss/FBt3/assignment_loss‚0.26277978604193775 2loss/FBt4/assignment_loss‚0.032932393566163655 .loss/FBt3/detection_loss‚2.946673905849457 trainer/global_step‚3499 _step‚285 /loss/FRt3/detection_loss‚1.7962283795249874 /loss/FRt4/detection_loss‚1.7510328077561488 /loss/FBt4/detection_loss‚0.7146601418414747 0loss/FRt3/assignment_loss‚1.6952179207884033 /loss/FBt1/detection_loss‚2.3615071773529053ƒ}ÆÒcÞÝ 2loss/SRqqt1/assignment_loss‚1.6177869649095968 1loss/SRqqt2/assignment_loss‚2.123323903977871 /loss/FRt1/detection_loss‚3.0054895904931156 /loss/FBt2/detection_loss‚2.5332283865321767 _runtime‚5887.160260937 0loss/FRt2/assignment_loss‚3.5722237672561254 2loss/SRqqt3/assignment_loss‚1.2153521278159363 0loss/FBt1/assignment_loss‚0.2762899082484864 1loss/SRqqt3/detection_loss‚1.1025548860709244 % _timestamp‚1.7885266489397187e+09 2loss/SRqqt4/assignment_loss‚1.8803740479052067 1loss/SRqqt1/detection_loss‚2.0851360214027492 /loss/SRqqt2/detection_loss‚2.80782699684302 0loss/SRqqt4/detection_loss‚3.310701002677282 0loss/FRt4/assignment_loss‚2.4913226536342075 1loss/FBt2/assignment_loss‚0.09638076117135018 1loss/FBt3/assignment_loss‚0.26277978604193775 2loss/FBt4/assignment_loss‚0.032932393566163655 .loss/FBt3/detection_loss‚2.946673905849457 trainer/global_step‚3499 _step‚285 /loss/FRt3/detection_loss‚1.7962283795249874 /loss/FRt4/detection_loss‚1.7510328077561488 /loss/FBt4/detection_loss‚0.7146601418414747 0loss/FRt3/assignment_loss‚1.6952179207884033 /loss/FBt1/detection_loss‚2.3615071773529053 epoch‚2 %loss/total_loss‚46.12060969501472 /loss/FRt2/detection_loss‚2.9099364907313614 0loss/FRt1/assignment_loss‚3.5316496726238364»ÇÀ‘ß‚(:† À€ëÔ–ˆŽù gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚132.183' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.4366 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0&w~l}à‚(:ò À€ëÔ¶æ¤ú disk.sda3.in‚0 disk.sda3.out‚ 168.76953125 disk.nvme0n1.in‚ 13805.484375) disk./.usagePercent‚78.98323428162894 disk.drbd1260.out‚ 0.80859375" disk.nvme0c0n1.in‚ 13805.484375 disk.sda1.out‚0! proc.memory.rssMB‚ 2836.296875 disk.dm-1.in‚0 disk.sda2.in‚0) disk.nvme0c0n1.out‚110883.19482421875' disk.nvme0n1.out‚110883.19482421875 disk.sda1.in‚0* proc.memory.percent‚17.311382293701172 cpu‚25.34453588783405# disk./.usageGB‚1176.90726852417 disk.drbd1260.in‚0 disk.dm-0.in‚ 13805.484375 disk.sda.in‚0 disk.sda.out‚ 168.76953125$ memory_percent‚98.39258193969727& proc.memory.availableMB‚ 263.359375 proc.cpu.threads‚17 network.sent‚ 8.822163e+06 network.recv‚ 1.895069e+06 disk.dm-1.out‚ 168.76953125 disk.sda2.out‚0$ disk.dm-0.out‚110883.19482421875ÜxŸá‚(:… Ï€ëÔÐù gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚95.023' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.0046 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0”QÍT€â‚(:õ Ï€ëÔ¯ñãû disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.80859375) disk.nvme0c0n1.out‚111320.44482421875* proc.memory.percent‚17.311382293701172$ disk.dm-0.out‚111320.44482421875 disk.sda.out‚ 169.17578125 cpu‚25.34428469838749 proc.cpu.threads‚17 network.sent‚ 8.929408e+06% disk./.usageGB‚1176.9072494506836 disk.dm-1.out‚ 169.17578125 disk.sda2.in‚0 disk.sda3.out‚ 169.17578125 disk.nvme0n1.in‚ 13911.984375' proc.memory.availableMB‚ 263.8671875! proc.memory.rssMB‚ 2836.296875 network.recv‚ 1.900312e+06) disk./.usagePercent‚78.98323300159126 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda3.in‚0" disk.nvme0c0n1.in‚ 13911.984375' disk.nvme0n1.out‚111320.44482421875$ memory_percent‚98.38948249816895 disk.sda2.out‚0 disk.dm-0.in‚ 13911.984375 disk.sda1.in‚0 disk.sda1.out‚0‘P×{ã‚(:ð Þ€ëÔúØÌø disk.sda1.out‚0 disk.sda2.in‚0$ memory_percent‚98.38950634002686* proc.memory.percent‚17.311382293701172 cpu‚25.344034362908076% disk./.usageGB‚1176.8837814331055 disk.sda.out‚ 169.49609375& disk.nvme0n1.out‚111719.5263671875 disk.sda1.in‚0 disk.sda2.out‚0 network.sent‚ 8.933761e+06 network.recv‚ 1.903134e+06) disk./.usagePercent‚78.98165804324061 disk.dm-0.in‚ 13911.984375# disk.dm-0.out‚111719.5263671875 disk.sda.in‚0 disk.sda3.out‚ 169.49609375" disk.nvme0c0n1.in‚ 13911.984375 disk.sda3.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 169.49609375 disk.drbd1260.in‚0 disk.drbd1260.out‚0.8125( proc.memory.availableMB‚ 263.86328125! proc.memory.rssMB‚ 2836.296875 proc.cpu.threads‚17( disk.nvme0c0n1.out‚111719.5263671875 disk.nvme0n1.in‚ 13911.984375Rq}Бä‚(:† Þ€ëÔ½þáø gpu.0.gpu‚10.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚104.131' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.8262 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÕÂ),‘å‚(:† í€ëÔËÀþö gpu.0.gpu‚23.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚103.361' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.6722 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0m[›Øåæ‚(:ì í€ëԘюø( disk.nvme0c0n1.out‚112094.4873046875 disk.sda1.in‚0 disk.drbd1260.out‚0.8125# disk.dm-0.out‚112094.4873046875( proc.memory.availableMB‚ 263.60546875* proc.memory.percent‚17.3113œ%ë’82293701172 disk.nvme0n1.in‚ 13911.984375& disk.nvme0n1.out‚112094.4873046875 disk.drbd1260.in‚0 disk.sda.out‚ 169.9296875 disk.sda3.out‚ 169.9296875! proc.memory.rssMB‚ 2836.296875 cpu‚25.34384492409859) disk./.usagePercent‚78.98323300159126 disk.dm-1.out‚ 169.9296875 disk.dm-0.in‚ 13911.984375 disk.sda.in‚0 disk.sda3.in‚0$ memory_percent‚98.39107990264893 network.recv‚ 1.907478e+06% disk./.usageGB‚1176.9072494506836" disk.nvme0c0n1.in‚ 13911.984375 disk.sda1.out‚0 disk.dm-1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 proc.cpu.threads‚17 network.sent‚ 8.939622e+06ÌÇ5ç‚(:… ü€ëÔŽÊž÷ gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚113.135' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.627 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0g ªsè‚(:è ü€ëÔŽ‡»ù cpu‚25.343307387890164 proc.cpu.threads‚17% disk./.usageGB‚1176.1977882385254 disk.sda1.in‚0 disk.dm-0.in‚ 13911.984375 disk.dm-0.out‚ 112273.828125 disk.sda3.out‚ 170.37890625* proc.memory.percent‚17.311382293701172) disk./.usagePercent‚78.93562046436571$ disk.nvme0c0n1.out‚ 112273.828125 disk.nvme0n1.in‚ 13911.984375" disk.nvme0n1.out‚ 112273.828125 disk.drbd1260.in‚0 disk.sda.out‚ 170.37890625 disk.sda3.in‚0$ memory_percent‚98.38969707489014 network.sent‚ 8.945861e+06" disk.nvme0c0n1.in‚ 13911.984375 disk.dm-1.in‚0 disk.dm-1.out‚ 170.37890625 disk.drbd1260.out‚ 0.81640625 disk.sda2.in‚0 disk.sda.in‚0( proc.memory.availableMB‚ 263.83203125 network.recv‚ 1.912506e+06 disk.sda1.out‚0 disk.sda2.out‚0! proc.memory.rssMB‚ 2836.296875å/HÄ2é- _wandbruntime‚5960  _runtime‚5960%ë Åê¿ % lr-AdamW/pg1‚0.000817953403092881 % lr-AdamW/pg2‚0.000817953403092881 trainer/global_step‚3549 % _timestamp‚1.7885267218580291e+09 _runtime‚5960.077483877 _step‚286¯…p{ÊëÄ trainer/global_step‚3549 % _timestamp‚1.7885267218580291e+09 _runtime‚5960.077483877 _step‚286 % lr-AdamW/pg1‚0.000817953403092881 % lr-AdamW/pg2‚0.000817953403092881žy¶ã2ì- _wandbruntime‚5960  _runtime‚5960ãåUÜ'íÖO M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚65196779 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚158974440958976 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚143478936339699 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059453 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25369029 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284328448 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚136466890690032 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 162935563 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268544 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7032216821760 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚143271440081920 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚136466873650672 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚96135035 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25369729 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚66826871 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚150303423235584 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚143479182360139 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚65197481 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚66826873 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚66826873 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚143271440081920 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 162961908 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚143271458084864 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 trainer/global_step‚3549 % _timestamp‚1.7885267218590431e+09 _runtime‚5960.078837799 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚150303674906624 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚700 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚39827752 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚39827750 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32248 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7153951202304 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚143271458084864 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚96135035 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚151820774085120 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚150303423235584 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671040 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚702 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7153666873856 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7012062689027 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚96108692 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚150303674906624 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7012291670107 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7031983153664 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚66826871 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32248 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7153666873856 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7012062689027 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚96108692 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚150303674906624 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020440 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7012291670107 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7031983153664 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚66826871 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚150303423235584 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚143479182360139 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚65197481 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚66826873 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚66826873 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900224 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚143271440081920 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 162961908 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981080 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288773 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚143271458084864 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26343 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 trainer/global_step‚3549 % _timestamp‚1.7885267218590431e+09 _runtime‚5960.078837799 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26345 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668096 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚150303674906624 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚700 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0ŸÝÊC2ï- _wandbruntime‚5961  _runtime‚5961½Â<&ðßM EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚66845148 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 163006901 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚96135384 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 163006901 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚39838554 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 318740992 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7155952545792 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚66845146 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚650 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚159018115800064 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚143518671206757 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 _step‚288 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚96135384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚66845148 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 162980530 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚150344673298432 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚39838552 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚150344924982784 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚151862458055168 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65214647 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚65215299 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚150344924982784 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25376745 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7014250159301 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7034180860928 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚66845146 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_æU©pool.freed‚ 7033947179520 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0¹å‹1ê&ñäM BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚150344673298432 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚39838552 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚150344924982784 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚151862458055168 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65214647 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚65215299 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚150344924982784 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25376745 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7014250159301 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7034180860928 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚66845146 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7033947179520 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚66845148 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 163006901 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚96135384 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 163006901 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚39838554 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 318740992 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7155952545792 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚66845146 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚650 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚159018115800064 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚143518671206757 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 _step‚288 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚96135384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚66845148 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 162980530 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 "âé¥2ò- _wandbruntime‚5961  _runtime‚5961•»ŠYóÓ .loss/FRt1/detection_loss‚3.010037982038089 1loss/SRqqt1/detection_loss‚2.1182366216436344 0loss/SRqqt4/detection_loss‚2.632606174945831 trainer/global_step‚3549 0loss/FRt4/assignment_loss‚3.3515248216834723 2loss/SRqqt3/assignment_loss‚1.0696045088950756 1loss/SRqqt3/detection_loss‚0.9847247522008026 _step‚289 /loss/FRt1/assignment_loss‚3.114784083196095 2loss/SRqqt1/assignment_loss‚1.5449965113892834 /loss/FRt2/detection_loss‚3.4484546706080437 _runtime‚5961.025564069 /loss/FRt3/detection_loss‚1.5657375968164868 .loss/FBt3/detection_loss‚2.304262121518453 .loss/FBt4/detection_loss‚0.738281026671757 0loss/FBt1/assignment_loss‚0.6076315951453413 0loss/FBt2/assignment_loss‚0.4396005271841302 1loss/FBt4/assignment_loss‚0.16676654147840608 0loss/SRqqt2/detection_loss‚2.390407601992289 /loss/FBt2/detection_loss‚2.3315210001809255 &loss/total_loss‚45.521449604621814 epoch‚2 % _timestamp‚1.7885267228061697e+09 1loss/FBt3/assignment_loss‚0.25608031885672566 /loss/FBt1/detection_loss‚1.9279660049237703 /loss/FRt3/assignment_loss‚1.907127783412025 /loss/FRt4/detection_loss‚1.8611792908460998 2loss/SRqqt4/assignment_loss‚2.2279743361473083 /loss/FRt2/assignment_loss‚3.561064804593722 1loss/SRqqt2/assignment_loss‚1.960878928254048MGCk^ôØ /loss/FRt3/assignment_loss‚1.907127783412025 /loss/FRt4/detection_loss‚1.8611792908460998 2loss/SRqqt4/assignment_loss‚2.2279743361473083 /loss/FRt2/assignment_loss‚3.561064804593722 1loss/SRqqt2/assignment_loss‚1.960878928254048 .loss/FRt1/detection_loss‚3.010037982038089 1loss/SRqqt1/detection_loss‚2.1182366216436344 0loss/SRqqt4/detection_loss‚2.632606174945831 trainer/global_step‚3549 0loss/FRt4/assignment_loss‚3.3515248216834723 2loss/SRqqt3/assignment_loss‚1.0696045088950756 1loss/SRqqt3/detection_loss‚0.9847247522008026 _step‚289 /loss/FRt1/assignment_loss‚3.114784083196095 2loss/SRqqt1/assignment_loss‚1.5449965113892834 /loss/FRt2/detection_loss‚3.4484546706080437 _runtime‚5961.025564069 /loss/FRt3/detection_loss‚1.5657375968164868 .loss/FBt3/detection_loss‚2.304262121518453 .loss/FBt4/detection_loss‚0.738281026671757 0loss/FBt1/assignment_loss‚0.6076315951453413 0loss/FBt2/assignment_loss‚0.4396005271841302 1loss/FBt4/assignment_loss‚0.16676654147840608 0loss/SRqqt2/detection_loss‚2.390407601992289 /loss/FBt2/detection_loss‚2.3315210001809255 &loss/total_loss‚45.521449604621814 epoch‚2 % _timestamp‚1.7885267228061697e+09 1loss/FBt3/assignment_loss‚0.25608031885672566 /loss/FBt1/detection_loss‚1.9279660049237703¡yJz‘õ‚(:† ‹ëÔ±¤’÷ gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚125.058' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.0116 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚04f;?wö‚(:ì ‹ëÔÝÛ’ù network.sent‚ 8.950926e+06) disk./.usagePercent‚78.93572005129653 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.81640625 disk.sda3.out‚ 171.2109375& disk.nvme0c0n1.out‚112610.46484375 disk.sda1.in‚0 cpu‚25.34270416677596 network.recv‚ 1.91628e+06 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.sda.in‚0 disk.sda.out‚ 171.2109375$ memory_percent‚98.39131832122803* proc.memory.percent‚17.311382293701172 proc.cpu.threads‚17% disk./.usageGB‚1176.1992721557617 disk.dm-0.in‚ 14056.6171875! disk.dm-0.out‚112610.46484375 disk.sda3.in‚0# disk.nvme0c0n1.in‚ 14056.6171875! disk.nvme0n1.in‚ 14056.6171875 disk.sda1.out‚0( proc.memory.availableMB‚ 263.56640625 disk.dm-1.out‚ 171.2109375 disk.sda2.out‚0$ disk.nvme0n1.out‚112610.46484375! proc.memory.rssMB‚ 2836.296875ถ‘÷‚(:† šëÔ哳ö gpu.0.gpu‚25.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚105.363' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.0726 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Y!ÈX‚ø‚(:÷ šëÔ®áà÷ network.recv‚ 1.922176e+06 disk.sda2.in‚0 disk.sda2.out‚0$ disk.dm-0.out‚112707.25830078125 disk.sda.out‚ 171.73828125 disk.sda3.in‚0 disk.sda3.out‚ 171.73828125 network.sent‚ 9.060381e+06) disk./.usagePercent‚78.93572312338695$ disk./.usageGB‚1176.199317932129' disk.nvme0n1.out‚112707.25830078125 disk.sda1.in‚0 disk.dm-1.out‚ 171.73828125 disk.drbd1260.in‚0' proc.memory.availableMB‚ 263.5390625) disk.nvme0c0n1.out‚112707.25830078125 disk.sda.in‚0! disk.nvme0n1.in‚ 14056.6171875 disk.dm-1.in‚0! proc.memory.rssMB‚ 2836.296875 proc.cpu.threads‚17 disk.dm-0.in‚ 14056.6171875 disk.sda1.out‚0 disk.drbd1260.out‚ 0.82421875# memory_percent‚98.3914852142334* proc.memory.percent‚17.311382293701172 cpu‚25.342154448828452# disk.nvme0c0n1.in‚ 14056.6171875"bµÀ›ù‚(: ©ëÔÈïðö gpu.0.gpu‚22.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.564' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚18.712799999999998 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0áÆð©‚ú‚(:÷ ©ëÔÈû÷# disk.nvme0c0n1.in‚ 14056.6171875) disk.nvme0c0n1.out‚113086.58251953125 disk.dm-1.in‚0 disk.sda2.in‚0$ disk.dm-0.out‚113086.58251953125 disk.sda.in‚0 disk.sda.out‚ 172.08203125% disk./.usageGB‚1176.1992645263672 disk.sda2.out‚0 disk.sda3.out‚ 172.08203125% proc.memory.availableMB‚ 263.78125* proc.memory.percent‚17.311382293701172 proc.cpu.threads‚17! disk.nvme0n1.in‚ 14056.6171875 disk.sda1.in‚0 disk.dm-1.out‚ 172.08203125 disk.drbd1260.out‚ 0.82421875 disk.sda3.in‚0 cpu‚25.341552383439364 network.sent‚ 9.064324e+06) disk./.usagePercent‚78.93571953928146' disk.nvme0n1.out‚113086.58251953125 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.dm-0.in‚ 14056.6171875$ memory_percent‚98.39000701904297! proc.memory.rssMB‚ 2836.296875 network.recv‚ 1.926617e+06K)Æå‘û‚(:† ¸ëԲ݅ø gpu.0.gpu‚18.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚105.192' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.0384 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Mzžvü‚(:ë ¸ëÔ‘½ìù) disk.nvme0c0n1.out‚113326.06787109375! disk.nvme0n1.in‚ 14056.6171875$ disk.dm-0.out‚113326.06787109375 disk.sda.in‚0! proc.memory.rssMB‚ 2836.296875* proc.memory.percent‚17.311382293701172 network.sent‚ 9.070568e+06) disk./.usagePercent‚78.93414458093082 disk.sda3.out‚ 172.65625# disk.nvme0c0n1.in‚ 14056.6171875 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚0.828125 network.recv‚ 1.931665e+06 disk.dm-1.out‚ 172.65625 disk.sda2.in‚0 disk.dm-0.in‚ 14056.6171875 disk.sda.out‚ 172.65625# memory_percent‚98.3914852142334' proc.memory.availableMB‚ 263.5390625 proc.cpu.threads‚17$ disk./.usageGB‚1176.175796508789 disk.sda3.in‚0' disk.nvme0n1.out‚113326.06787109375 disk.dm-1.in‚0 disk.sda2.out‚0 cpu‚25.34163960655634tDûö2ý- _wandbruntime‚6024  _runtime‚6024ðºù6ÇþÁ & lr-AdamW/pg1‚0.0008137683643400616 & lr-AdamW/pg2‚0.0008137683643400616 trainer/global_step‚3599 % _timestamp‚1.7885267865648332e+09 _runtime‚6024.784440348 _step‚290ZcsÌÿÆ & lr-AdamW/pg1‚0.0008137683643400616 & lr-AdamW/pg2‚0.0008137683643400616 trainer/global_step‚3599 % _timestamp‚1.7885267865648332e+09 _runtime‚6024.784440348 _step‚290¢ÛEÛÈ2€- _wandbruntime‚6024  _runtime‚6024놇?Û'ÕO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚152365940791808 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚153904972587520 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7253799782912 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 165187265 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚138343408561232 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚67740623 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7130432528384 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 165213610 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚145235759934464 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚161158481750528 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚145453638271847 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚145235741931520 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚723 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚67740621 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚97472987 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚152366192462848 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚66086319 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 314560000 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚152365940791808 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 290619904 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0¬Qå¬î‚ÚO `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚145235759934464 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚161158481750528 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚145453638271847 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671040 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025280 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 339145216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚145235741931520 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26343 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚723 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚67740621 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚97472987 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚152366192462848 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900224 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26345 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚66086319 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 314560000 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025280 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚152365940791808 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 290619904 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 315205120 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚152365940791808 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚153904972587520 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7253799782912 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 165187265 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32005 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚138343408561232 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚67740623 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7130432528384 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 165213610 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32005 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668096 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0£9"2ƒ- _wandbruntime‚6025  _runtime‚6025•° æ&„àM 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚145275027968512 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 165258603 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 % _timestamp‚1.7885267874770815e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 165258603 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚152407442539008 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7132162886144 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚641 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310352384 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚138380921879296 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚145493127118465 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7111959218521 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚152407442539008 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚145275027968512 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚67758898 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7255789027840 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 trainer/global_step‚3599 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚67758896 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚40378654 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚153946656557568 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚161202159173120 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 165232232 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚145275045971456 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚97473336 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25725007 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚67758898 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 165232232 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7132396567552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚66103661 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7255502615552 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚152407190854656 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66103020 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286412288 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7112188199809 _step‚292 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚97473336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚161202469525504 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚138380938918656 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚67758896 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚97499705 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚145275045971456 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚40378652 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36Ý_Z>ë&…åM \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7132396567552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 315205120 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059453 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚66103661 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7255502615552 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚152407190854656 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66103020 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288773 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900224 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286412288 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7112188199809 _step‚292 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚97473336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025280 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26371 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚161202469525504 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚138380938918656 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚67758896 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚97499705 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚145275045971456 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚40378652 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684352 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚145275027968512 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 165258603 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 % _timestamp‚1.7885267874770815e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 165258603 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚152407442539008 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981288 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681408 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7132162886144 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚641 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310352384 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚138380921879296 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚145493127118465 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7111959218521 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚152407442539008 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚145275027968512 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚67758898 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7255789027840 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681408 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684352 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 trainer/global_step‚3599 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26369 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚67758896 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚40378654 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32005 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚153946656557568 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚161202159173120 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900224 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 165232232 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚145275045971456 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025280 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚97473336 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25725007 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32248 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚67758898 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 165232232¤ã ‰O2†- _wandbruntime‚6025  _runtime‚6025ßqܽ[‡Õ 0loss/SRqqt3/detection_loss‚1.139112891855641 /loss/FBt3/detection_loss‚2.3105678473200117 2loss/SRqqt1/assignment_loss‚1.6085871398753049 1loss/FBt1/assignment_loss‚0.26624644959735055 /loss/FRt2/detection_loss‚3.3586409254507585 1loss/SRqqt1/detection_loss‚1.7722210567824694 1loss/SRqqt2/detection_loss‚2.9062532608707747 % _timestamp‚1.7885267874793296e+09 2loss/SRqqt4/assignment_loss‚2.2554141993706045 0loss/FBt4/assignment_loss‚0.2777630870443239 %loss/total_loss‚46.26892374978815 0loss/FRt3/assignment_loss‚1.4448511479077515 1loss/FBt2/assignment_loss‚0.40917527929579217 .loss/FRt4/detection_loss‚1.721381090693818 .loss/FBt1/detection_loss‚2.874500036239624 0loss/FRt1/assignment_loss‚3.7319340362693323 /loss/FRt1/detection_loss‚2.7532776774782124 /loss/FBt2/detection_loss‚2.3444570953195747 epoch‚2 trainer/global_step‚3599 _step‚293 -loss/FRt3/detection_loss‚1.87468389145754 0loss/SRqqt4/detection_loss‚2.692951248242305 /loss/FRt2/assignment_loss‚3.845299769531597 0loss/FRt4/assignment_loss‚2.7051358048258156 1loss/FBt3/assignment_loss‚0.28508141321084884 /loss/FBt4/detection_loss‚0.7992378456725014 _runtime‚6025.698609898 2loss/SRqqt2/assignment_loss‚1.8593184081837535 2loss/SRqqt3/assignment_loss‚1.0328321472924447¡&0`ˆÚ 2loss/SRqqt1/assignment_loss‚1.6085871398753049 1loss/FBt1/assignment_loss‚0.26624644959735055 /loss/FRt2/detection_loss‚3.3586409254507585 1loss/SRqqt1/detection_loss‚1.7722210567824694 1loss/SRqqt2/detection_loss‚2.9062532608707747 % _timestamp‚1.7885267874793296e+09 2loss/SRqqt4/assignment_loss‚2.2554141993706045 0loss/FBt4/assignment_loss‚0.2777630870443239 %loss/total_loss‚46.26892374978815 0loss/FRt3/assignment_loss‚1.4448511479077515 1loss/FBt2/assignment_loss‚0.40917527929579217 .loss/FRt4/detection_loss‚1.721381090693818 .loss/FBt1/detection_loss‚2.874500036239624 0loss/FRt1/assignment_loss‚3.7319340362693323 /loss/FRt1/detection_loss‚2.7532776774782124 /loss/FBt2/detection_loss‚2.3444570953195747 epoch‚2 trainer/global_step‚3599 _step‚293 -loss/FRt3/detection_loss‚1.87468389145754 0loss/SRqqt4/detection_loss‚2.692951248242305 /loss/FRt2/assignment_loss‚3.845299769531597 0loss/FRt4/assignment_loss‚2.7051358048258156 1loss/FBt3/assignment_loss‚0.28508141321084884 /loss/FBt4/detection_loss‚0.7992378456725014 _runtime‚6025.698609898 2loss/SRqqt2/assignment_loss‚1.8593184081837535 2loss/SRqqt3/assignment_loss‚1.0328321472924447 0loss/SRqqt3/detection_loss‚1.139112891855641 /loss/FBt3/detection_loss‚2.3105678473200117¥Us`nœ‰‚(:‘ ÇëÔ££â÷ gpu.0.gpu‚34.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚33.0 gpu.0.powerWatts‚140.424' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.084799999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0{µ}Š‚(:ò ÇëÔ÷ï±ø) disk./.usagePercent‚78.92901214188437 disk.sda.in‚0 disk.sda.out‚ 173.1328125$ disk.nvme0c0n1.in‚14250.41015625 disk.sda1.out‚0 disk.drbd1260.out‚0.828125 disk.sda2.out‚0 disk.dm-0.in‚14250.41015625 network.recv‚ 1.935268e+06% disk./.usageGB‚1176.0993194580078 disk.sda3.out‚ 173.1328125) disk.nvme0c0n1.out‚113519.65771484375' disk.nvme0n1.out‚113519.65771484375 disk.sda1.in‚0 disk.sda2.in‚0$ disk.dm-0.out‚113519.65771484375 network.sent‚ 9.075554e+06" disk.nvme0n1.in‚14250.41015625 disk.dm-1.out‚ 173.1328125$ memory_percent‚98.39019775390625! proc.memory.rssMB‚ 2836.296875* proc.memory.percent‚17.311382293701172 disk.sda3.in‚0 disk.dm-1.in‚0 disk.drbd1260.in‚0" proc.memory.availableMB‚263.75 cpu‚25.341055916166773 proc.cpu.threads‚17Ä:'œ‹‚(:‘ ÖëÔíï•ú gpu.0.gpu‚21.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚120.493' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.098599999999998 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[úúyŒ‚(:î ÖëÔÿðçÿ network.recv‚ 1.940936e+06% disk./.usageGB‚1176.2571296691895 disk.dm-0.out‚114032.1328125 disk.sda3.out‚ 173.5703125$ disk.nvme0c0n1.in‚14250.41015625$ memory_percent‚98.39174747467041 cpu‚25.340521712785755 proc.cpu.threads‚17 network.sent‚ 9.183264e+06 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.83203125 disk.sda2.in‚0 disk.dm-0.in‚14250.41015625 disk.sda3.in‚0% disk.nvme0c0n1.out‚114032.1328125" disk.nvme0n1.in‚14250.41015625 disk.sda.in‚0 disk.sda.out‚ 173.5703125 disk.sda1.in‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 263.49609375! proc.memory.rssMB‚ 2836.296875* proc.memory.percent‚17.311382293701172) disk./.usagePercent‚78.93960291756837 disk.sda2.out‚0# disk.nvme0n1.out‚114032.1328125 disk.dm-1.in‚0 disk.dm-1.out‚ 173.5703125ÿìàu‚(:… åëÔĆš÷ gpu.0.gpu‚12.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚92.744' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.5488 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0²F¦âŽ‚(:é åëÔ¸‡ø disk.sda3.in‚0$ memory_percent‚98.39024543762207 cpu‚25.34016125936849% disk.nvme0c0n1.out‚114185.1328125 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.dm-0.in‚14250.41015625 disk.dm-0.out‚114185.1328125 disk.sda.in‚0 disk.sda3.out‚ 174.015625! proc.memory.rssMB‚ 2836.296875 network.sent‚ 9.189089e+06 network.recv‚ 1.945345e+06) disk./.usagePercent‚78.93958064491288% disk./.usageGB‚1176.2567977905273 disk.sda1.in‚0* proc.>ÂS‹omemory.percent‚17.311382293701172$ disk.nvme0c0n1.in‚14250.41015625" disk.nvme0n1.in‚14250.41015625# disk.nvme0n1.out‚114185.1328125 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0' proc.memory.availableMB‚ 263.7421875 proc.cpu.threads‚17 disk.dm-1.out‚ 174.015625 disk.drbd1260.out‚ 0.83203125 disk.sda.out‚ 174.015625“Û¹µ‡‚(:ü ôëÔž²¦÷$ memory_percent‚98.95656108856201( proc.memory.availableMB‚ 170.95703125* proc.memory.percent‚17.876100540161133) disk./.usagePercent‚78.94010546035807$ disk.nvme0c0n1.in‚14250.41015625' disk.nvme0n1.out‚114405.32958984375 disk.sda1.out‚0 disk.sda2.in‚0$ disk.dm-0.out‚114405.32958984375 disk.sda.out‚ 174.41015625 disk.sda3.out‚ 174.41015625 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.8359375 disk.sda.in‚0" proc.memory.rssMB‚ 2928.8203125 cpu‚25.34284502923813 proc.cpu.threads‚17 network.recv‚ 1.950544e+06% disk./.usageGB‚1176.2646179199219) disk.nvme0c0n1.out‚114405.32958984375" disk.nvme0n1.in‚14250.41015625 disk.dm-1.in‚0 disk.dm-1.out‚ 174.41015625 disk.sda2.out‚0 disk.sda3.in‚0 network.sent‚ 9.195862e+06 disk.sda1.in‚0 disk.dm-0.in‚14250.41015625 wÍ뜂(:‘ ôëÔâ™ò÷ gpu.0.gpu‚42.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚144.221' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.844200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0W­­°‘‚(:„ ƒ‚ëÔ¿¹Âø gpu.0.gpu‚2.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚92.852' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.5704 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0i±ó„’‚(:ù ƒ‚ëÔÃÁøû disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 174.91796875 disk.sda2.in‚0 disk.dm-0.in‚ 14356.8671875% disk./.usageGB‚1176.2528076171875# disk.nvme0c0n1.in‚ 14356.8671875 disk.dm-1.in‚0 disk.drbd1260.in‚0$ disk.dm-0.out‚114938.37646484375 disk.sda3.in‚0 disk.sda3.out‚ 174.91796875( proc.memory.availableMB‚ 232.04296875 network.recv‚ 1.954092e+06 disk.drbd1260.out‚ 0.8359375 disk.sda.in‚0$ memory_percent‚98.58372211456299* proc.memory.percent‚17.504215240478516 proc.cpu.threads‚17) disk./.usagePercent‚78.93931286103206) disk.nvme0c0n1.out‚114938.37646484375 disk.sda2.out‚0 disk.sda.out‚ 174.91796875! proc.memory.rssMB‚ 2867.890625 cpu‚25.351941798439793 network.sent‚ 9.200822e+06! disk.nvme0n1.in‚ 14356.8671875' disk.nvme0n1.out‚114938.37646484375×`Ø“‚(:ô ’‚ëÔÖý÷ disk.sda2.out‚0 disk.sda3.in‚0 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚98.91738891601562# proc.memory.rssMB‚ 2922.02734375$ disk./.usageGB‚1176.327262878418 disk.dm-0.in‚14382.58203125$ disk.nvme0c0n1.in‚14382.58203125) proc.memory.percent‚17.83463954925537 cpu‚25.36106713269627 disk.sda3.out‚ 175.3828125 network.recv‚ 1.957933e+06) disk./.usagePercent‚78.94430961608536$ disk.dm-0.out‚115222.05712890625 disk.sda.in‚0 disk.sda.out‚ 175.3828125" disk.nvme0n1.in‚14382.58203125' disk.nvme0n1.out‚115222.05712890625 network.sent‚ 9.206159e+06 disk.dm-1.in‚0 disk.dm-1.out‚ 175.3828125 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.83984375# proc.memory.availableMB‚177.375 proc.cpu.threads‚17) disk.nvme0c0n1.out‚115222.05712890625 disk.sda2.in‚0Ìgþœ”‚(:‘ ’‚ëÔëÏ’ù gpu.0.gpu‚30.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚107.645' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.528999999999996 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ª¿•‚(:„ ¡‚ëÔ§¹Èù gpu.0.gpu‚21.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚103.24' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.648 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£vrv‰–‚(:þ ¡‚ëÔˆÌêû# proc.memory.rssMB‚ 2931.07421875 network.sent‚ 9.214022e+06% disk./.usageGB‚1176.3894958496094 disk.dm-0.in‚14382.58203125" disk.nvme0n1.in‚14382.58203125 disk.sda1.out‚0' proc.memory.availableMB‚ 168.0859375* proc.memory.percent‚17.889857292175293 cpu‚25.368988336872444 network.recv‚ 1.962668e+06 disk.sda.in‚0 disk.sda3.out‚ 176.03515625$ memory_percent‚98.97408485412598 proc.cpu.threads‚17) disk./.usagePercent‚78.94848612299896 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.83984375 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚0$ disk.nvme0c0n1.in‚14382.58203125' disk.nvme0n1.out‚115452.79931640625$ disk.dm-0.out‚115452.79931640625 disk.sda.out‚ 176.03515625) disk.nvme0c0n1.out‚115452.79931640625 disk.sda1.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 176.03515625Lð J‘—‚(:† °‚ëÔó׺ø gpu.0.gpu‚19.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚101.848' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.3696 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ðP«P„˜‚(:ù °‚ëÔÐôÈû network.sent‚ 9.218403e+06 network.recv‚ 1.967278e+06% disk./.usageGB‚1176.5144958496094$ disk.dm-0.out‚115731.90283203125' disk.nvme0n1.out‚115731.90283203125 disk.sda2.in‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 2931.07421875 disk.dm-0.in‚14382.58203125 disk.sda.out‚ 176.328125 disk.sda3.in‚0" disk.nvme0n1.in‚14382.58203125 disk.dm-1.out‚ 176.328125 disk.drbd1260.in‚0* proc.memory.percent‚17.889857292175293 proc.cpu.threads‚17 disk.sda.in‚0) disk.nvme0c0n1.out‚115731.90283203125 disk.sda1.in‚0 disk.dm-1.in‚0$ memory_percent‚98.97072315216064( proc.memory.availableMB‚ 168.63671875 cpu‚25.379249703258147) disk./.usagePercent‚78.95687497788096 disk.sda3.out‚ 176.328125$ disk.nvme0c0n1.in‚14382.58203125 disk.sda1.out‚0 disk.drbd1260.out‚ 0.84765625Ž ™‚(:„ ¿‚ëÔÕÅ÷ gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚29.0 gpu.0.powerWatts‚92.744' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.5488 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0M<¢Ýƒš‚(:ø ¿‚ëÔ±­ºù$ disk.nvme0c0n1.in‚14382.58203125" disk.nvme0n1.in‚14382.58203125 disk.sda1.out‚0 disk.dm-1.out‚ 176.859375 disk.drbd1260.in‚0 disk.sda2.in‚0 network.sent‚ 9.223492e+06 disk.dm-0.in‚14382.58203125$ disk.dm-0.out‚116001.15283203125 disk.sda3.out‚ 176.859375 disk.sda2.out‚0$ memory_percent‚99.08149242401123# proc.memory.rssMB‚ 2949.16015625 cpu‚25.38754180249093) disk./.usagePercent‚78.95674876616657 disk.sda.out‚ 176.859375 disk.drbd1260.out‚ 0.84765625( proc.memory.availableMB‚ 150.48828125 proc.cpu.threads‚17 disk.sda.in‚0 disk.sda3.in‚0) disk.nvme0c0n1.out‚116001.15283203125' disk.nvme0n1.out‚116001.15283203125 disk.sda1.in‚0 disk.dm-1.in‚0* proc.memory.percent‚18.000245094299316 network.recv‚ 1.971009e+06% disk./.usageGB‚1176.5126152038574D¥æ2›- _wandbruntime‚6152  _runtime‚6152E€X©B*œ¼T VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚67452310 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚391 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚273 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 167642657 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚140646681401072 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 7188054667739 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 167663369 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚41242724 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1300 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚147834719029451 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚147808969005056 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚156559873615360 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 7332959990784 BDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 YLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚98657328 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚309 UHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚98636618 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚32005 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚163892658321920 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚26209584 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 _runtime‚6152.853416307 QDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚98636618 XIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 7134511104 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚309 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚309 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 6458321920 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚582 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 7208488381952 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚20712 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 175284224 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚140646664361712 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 572522496 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 7208664193536 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 7134511104 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 193814528 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚36 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 347418624 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 572522496 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 _step‚294 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚147808969005056 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 167642657 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚273 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚69006039 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 7332808646656 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚147834907889279 $ _timestamp‚1.788526914632728e+09 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 6966900224 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3532ØF=lG*ÁT $ _timestamp‚1.788526914632728e+09 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 6966900224 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3532 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚67452310 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚391 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚273 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 167642657 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚140646681401072 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 7188054667739 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 167663369 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚41242724 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1300 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚147834719029451 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚147808969005056 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚156559873615360 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 7332959990784 BDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 YLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚98657328 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚309 UHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚98636618 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚32005 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚163892658321920 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚26209584 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 _runtime‚6152.853416307 QDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚98636618 XIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 7134511104 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚309 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚309 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 6458321920 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚582 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 7208488381952 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚20712 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 175284224 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚140646664361712 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 572522496 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 7208664193536 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 7134511104 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 193814528 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚36 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 347418624 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 572522496 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 _step‚294 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚147808969005056 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 167642657 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚273 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚69006039 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 7332808646656 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚147834907889279¦8ýT2ž- _wandbruntime‚6154  _runtime‚6154åLö¶– ŸÇR [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 171820468 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 560288773 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 572522496 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 167687659 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 175284224 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚273 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚32248 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚147839052682240 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚309 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 7209138053632 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚393 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚163923493935616 QBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 7134511104 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚20710 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 167666947 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚69021421 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 572522496 trainer/global_step‚3626 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32248 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚26215370 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚140672224011344 JADeviceStatsMonitor.on_validation_batc'¸°·h_end/active.large_pool.peak‚3532 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1319 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚36 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 6966900224 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚391 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚67469776 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚155047996921344 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 7208962242048 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 188859828 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 7188526875551 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 6741059453 _runtime‚6154.096013321 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚155048190735872 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 167666947 C;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚582 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 7188698696019 RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 6966900224 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚147860750886895 VGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 7134511104 TEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 7134511104 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚163923669219840 _step‚295 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚67470169 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚41254406 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚156590211428864 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚140672241050704 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚582 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1300 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚582 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚98666236 SFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚98645526 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚309 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3532 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 347418624 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚309 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚155048190735872 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚155047996921344 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20712 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 7208962242048 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚26215761 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚41254408 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚69021421 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚156590235368960 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚273 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 565025280 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚32005 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 7333282506752 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 193814528 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 6561988608 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚582 SFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚98666236 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 7333433850880 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 151344128 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚582 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚98666236 SFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚98645526 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚309 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3532 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 347418624 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚309 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚155048190735872 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚155047996921344 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20712 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 7208962242048 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚26215761 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚41254408 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚69021421 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚156590235368960 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚273 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 565025280 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚32005 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 7333282506752 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 193814528 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 6561988608 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚582 SFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚98666236 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 7333433850880 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 151344128 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32248 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚26215370 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚140672224011344§~$3‡¡‚(:ü ΂ëÔݬ…ø$ memory_percent‚98.58472347259521 proc.cpu.threads‚17 disk.sda1.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 177.22265625 disk.sda2.out‚0! proc.memory.rssMB‚ 2867.890625* proc.memory.percent‚17.504215240478516 cpu‚25.396472097595982 network.recv‚ 1.974659e+06) disk./.usagePercent‚78.97208054537715% disk./.usageGB‚1176.7410697937012" disk.nvme0n1.in‚14521.58203125 disk.sda2.in‚0 disk.dm-0.in‚14521.58203125 disk.sda.out‚ 177.22265625' disk.nvme0n1.out‚116376.51416015625 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.8515625 disk.sda.in‚0 disk.sda3.in‚0$ disk.nvme0c0n1.in‚14521.58203125( proc.memory.availableMB‚ 231.87890625 network.sent‚ 9.228603e+06 disk.sda1.out‚0$ disk.dm-0.out‚116376.51416015625 disk.sda3.out‚ 177.22265625) disk.nvme0c0n1.out‚116376.51416015625@1.M‘¢‚(:† ΂ëÔݘù gpu.0.gpu‚20.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚112.472' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.4944 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0óíR¸2£- _wandbruntime‚6177  _runtime‚6177®§Wƒ2¤©¨ ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 6validation_average_jet_accuracy‚0.4009475111961365 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.41166412830352783 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3231481611728668 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 9"Purity/0FRt2SRqqt2FBt/event_purity‚0.0714285746216774 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_6‚1 5 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.90666663646698 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.07886485010385513 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.4295981824398041 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.4204545319080353 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 *jet/accuracy_1_of_1‚0.4398961067199707 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 % _timestamp‚1.7885269391941707e+09 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.4329014718532562 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_2_of_7‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8888888955116272 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 particle/accuracy_7_of_7‚0.5 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.2083333283662796 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5606567859649658 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 jet/accuracy_5_of_6‚0 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9184398055076599 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.49008622765541077 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.19473683834075928 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9735631346702576 ;"Purity/2FRt1SRqqt2FBt/event_purity‚0.022727273404598236 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2172619104385376 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.2857142984867096 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 ;"Purity/1FRt2SRqqt1FBt/event_purity‚0.008620689623057842 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.12719297409057617 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.12627504765987396 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4166666865348816 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.23333333432674408 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9875340461730957 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.34375 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_3_of_6‚1 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.45313534140586853 +jet/accuracy_3_of_6‚0.10897436738014221 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 jet/accuracy_6_of_6‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.4285714328289032 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2597792446613312 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7878787517547607 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 particle/accuracy_1_of_1‚1 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2222222238779068 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.3641975522041321 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.19032208621501923 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9812804460525513 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 jet/accuracy_6_of_7‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.37979573011398315 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9728301763534546 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9579861164093018 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.16222135722637177 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.1160871759057045 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.15226337313652039 *jet/accuracy_2_of_4‚0.3333214521408081 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.14768347144126892 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1544138789176941 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 particle/accuracy_2_of_5‚1 ;"Purity/3FRt*SRqqt1FBt/event_purity‚0.008333333767950535 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1639704555273056 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.17526724934577942 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ;"Purity/2FRt0SRqqt1FBt/event_purity‚0.012648810632526875 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 particle/accuracy_1_of_5‚1 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.31429171562194824 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.3844049870967865 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_4_of_5‚1 8"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.675438642501831 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.492946982383728 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3670520484447479 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.43030044436454773 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4206608235836029 *jet/accuracy_1_of_6‚0.9807692170143127 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3390637934207916 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 9"Purity/0FRt1SRqqt3FBt/event_purity‚0.1428571492433548 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.32914531230926514 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9842999577522278 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 <"Purity/3FRt*SRqqt*FBt/event_purity‚0.0023419205099344254 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9994331002235413 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.2308429479598999 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.011185983195900917 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.1197916641831398 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_5_of_6‚1 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5833333134651184 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.4719374477863312 *jet/accuracy_1_of_4‚0.8761470317840576 ,jet/accuracy_4_of_5‚0.017249103635549545 particle/accuracy_1_of_3‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4163374602794647 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.035467538982629776 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.00394606776535511 ;"Purity/3FRt0SRqqt*FBt/event_purity‚0.011111111380159855 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.8500000238418579 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.14086180925369263 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9826807379722595 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.17272444069385529 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7727272510528564 ,particle/sensitivity‚0.28416207432746887 8"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.503333330154419 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.3927299380302429 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0020502645056694746 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9563491940498352 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4024397134780884 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.3933449685573578 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.22680792212486267 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.22145061194896698 particle/accuracy_6_of_6‚1 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.32941171526908875 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.2083333432674408 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 & Purity/·Y.ƒ­a*FRt3SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5917253494262695 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 & Purity/3FRt0SRqqt2FBt/FRt_purity‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1626984179019928 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.4538644850254059 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.27444660663604736 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3716356158256531 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 *jet/accuracy_3_of_4‚0.0628167986869812 (particle/accuracy‚0.8440755009651184 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.3499999940395355 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.18098396062850952 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5074579119682312 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5815476179122925 +"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.25 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 9"Purity/*FRt1SRqqt3FBt/event_purity‚0.1111111119389534 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.07894736528396606 particle/accuracy_2_of_2‚1 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 :"Purity/2FRt*SRqqt2FBt/event_purity‚0.05426356941461563 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ,jet/accuracy_4_of_4‚0.008785532787442207 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.24166668951511383 :"Purity/*FRt2SRqqt2FBt/event_purity‚0.03999999910593033 jet/accuracy_1_of_7‚1 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9966569542884827 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 6 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.903508722782135 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5134441256523132 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_5_of_7‚1 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6723325252532959 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 jet/accuracy_3_of_7‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.3294612765312195 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.013594469986855984 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4254404902458191 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.42527228593826294 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 *jet/accuracy_2_of_3‚0.2203972488641739 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.3951389491558075 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.25925925374031067 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9967953562736511 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.4056878983974457 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9990079402923584 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.46910926699638367 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9947887659072876 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.561768114566803 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9985569715499878 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.4757598042488098 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5708333253860474 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.020181190222501755 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5370370745658875 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 *jet/accuracy_1_of_5‚0.9336532950401306 *jet/accuracy_2_of_5‚0.4460829198360443 particle/accuracy_2_of_3‚1 particle/accuracy_2_of_4‚1 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5882353186607361 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5684332251548767 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.19879211485385895 ."Purity/3FRt0SRqqt1FBt/event_purity‚0.03125 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.8666666746139526 particle/accuracy_4_of_7‚1 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2253185212612152 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2069786936044693 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.31450676918029785 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.22640535235404968 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9879111051559448 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.27097058296203613 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.01867816224694252 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.3333333134651184 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.4099031090736389 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.3875800371170044 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.39730677008628845 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.15909090638160706 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9734195470809937 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3690873384475708 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.3448948860168457 particle/accuracy_5_of_5‚1 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2708333134651184 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5430651307106018 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.32576504349708557 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.04909916967153549 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 particle/accuracy_4_of_6‚1 jet/accuracy_4_of_7‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 9"Purity/*FRt*SRqqt3FBt/event_purity‚0.1666666716337204 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5789473652839661 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.59375 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.4437984526157379 ) Purity/*FRt2SRqqt2FBt/FRt_purity‚0.25 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9914966225624084 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.27081847190856934 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.46380388736724854 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.40666666626930237 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.16015219688415527 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.37768203020095825 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_3_of_3‚0.02473326586186886 (particle/f_score‚0.36256247758865356 7 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.1688888967037201 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.3958333432674408 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.1428571492433548 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8333333134651184 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.2083333432674408 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.1281268447637558 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.007395382504910231 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.4444444477558136 particle/accuracy_6_of_7‚1 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9992941617965698 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5514119863510132 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5924124717712402 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 *jet/accuracy_1_of_2‚0.6993187069892883 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.17559488117694855 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.1608092039823532 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.22721077501773834 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2222222238779068 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.1875 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 *loss/val_total_loss‚44.985870146357826 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5168235301971436 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 jet/accuracy_5_of_7‚0 epoch‚2 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.4894269108772278 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9871526956558228 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.37663692235946655 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 trainer/global_step‚3626 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9865419864654541 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.0946962982416153 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9887649416923523 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.04553823918104172 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9928156137466431 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 8"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.388525128364563 ) Purity/2FRt3SRqqt1FBt/FRt_purity‚0.25 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.19587139785289764 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.14814215898513794 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.17526724934577942 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0012210012646391988 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 7 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.2261742204427719 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.01737385243177414 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1805555671453476 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.32265791296958923 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5654162168502808 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9662452936172485 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.3499999940395355 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.2446969598531723 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.4727807343006134 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9963145852088928 8"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.359708309173584 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 _runtime‚ 6177.41638305 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8161837458610535 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.20642724633216858 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 -"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.1875 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.017808588221669197 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.16728423535823822 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.20845499634742737 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.09233281761407852 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 8"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.550000011920929 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.007980599999427795 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.15338543057441711 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.26508671045303345 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.39017853140830994 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.019098035991191864 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.37892061471939087 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.1537572592496872 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5691391229629517 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 +"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.75 6 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.977656364440918 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 particle/accuracy_2_of_6‚1 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1442434936761856 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.13748739659786224 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.18773946166038513 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9873205423355103 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.4614197313785553 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5374014973640442 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4021349251270294 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.37621235847473145 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.30944058299064636 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.18837787210941315 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.2953064739704132 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.27327582240104675 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 +jet/accuracy_3_of_5‚0.08455580472946167 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 +"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.25 jet/accuracy_7_of_7‚0 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.2083333283662796 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5287730097770691 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 8"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.601190447807312 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 particle/accuracy_3_of_3‚1 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.991875410079956 ,jet/accuracy_5_of_5‚0.001792114693671465 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9270833134651184 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 :"Purity/*FRt2SRqqt1FBt/event_purity‚0.01309523917734623 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.007056144531816244 particle/accuracy_3_of_4‚1 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9994331002235413 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.013139831833541393 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_7‚1 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.08916987478733063 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.21867816150188446 *jet/accuracy_1_of_3‚0.8185186982154846 particle/accuracy_4_of_4‚1 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9824561476707458 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4879385828971863 +particle/specificity‚0.9477013349533081 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 *jet/accuracy_2_of_6‚0.4551282227039337 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.3083333373069763 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4285714328289032 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9464285969734192 particle/accuracy_1_of_7‚1 _step‚296 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5207858681678772 8"Purity/0FRt0SRqqt2FBt/event_purity‚0.336351215839386 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0022675737272948027 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.988946259021759 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.2520270347595215 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 +jet/accuracy_4_of_6‚0.03846153989434242 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6363846659660339 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.4095490276813507 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.16531895101070404 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.4382183849811554 +"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.75 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.41215068101882935 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.22451160848140717 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.09751074016094208 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.2787524163722992 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2675339877605438 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.3333333134651184 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.42356598377227783 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.3933449685573578 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1666666716337204 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.932539701461792 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.21996353566646576 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9904747605323792 particle/accuracy_1_of_2‚1 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1805555671453476 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9908382296562195 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 particle/accuracy_3_of_7‚1 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0ÐÑ\E¥®¨ 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.4099031090736389 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.3875800371170044 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.39730677008628845 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.15909090638160706 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9734195470809937 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3690873384475708 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.3448948860168457 particle/accuracy_5_of_5‚1 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2708333134651184 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5430651307106018 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.32576504349708557 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.04909916967153549 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 particle/accuracy_4_of_6‚1 jet/accuracy_4_of_7‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 9"Purity/*FRt*SRqqt3FBt/event_purity‚0.1666666716337204 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5789473652839661 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.59375 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.4437984526157379 ) Purity/*FRt2SRqqt2FBt/FRt_purity‚0.25 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9914966225624084 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.27081847190856934 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.46380388736724854 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.40666666626930237 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.16015219688415527 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.37768203020095825 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_3_of_3‚0.02473326586186886 (particle/f_score‚0.36256247758865356 7 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.1688888967037201 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.3958333432674408 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.1428571492433548 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8333333134651184 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.2083333432674408 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.1281268447637558 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.007395382504910231 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.4444444477558136 particle/accuracy_6_of_7‚1 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9992941617965698 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5514119863510132 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5924124717712402 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 *jet/accuracy_1_of_2‚0.6993187069892883 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.17559488117694855 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.1608092039823532 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.22721077501773834 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2222222238779068 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.1875 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 *loss/val_total_loss‚44.985870146357826 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5168235301971436 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 jet/accuracy_5_of_7‚0 epoch‚2 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.4894269108772278 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9871526956558228 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.37663692235946655 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 trainer/global_step‚3626 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9865419864654541 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.0946962982416153 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9887649416923523 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.04553823918104172 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9928156137466431 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 8"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.388525128364563 ) Purity/2FRt3SRqqt1FBt/FRt_purity‚0.25 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.19587139785289764 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.14814215898513794 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.17526724934577942 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0012210012646391988 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 7 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.2261742204427719 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.01737385243177414 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1805555671453476 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.32265791296958923 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5654162168502808 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9662452936172485 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.3499999940395355 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.2446969598531723 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.4727807343006134 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9963145852088928 8"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.359708309173584 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 _runtime‚ 6177.41638305 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8161837458610535 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.20642724633216858 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 -"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.1875 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.2}p†¡ðu00397183420137e-05 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.017808588221669197 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.16728423535823822 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.20845499634742737 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.09233281761407852 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 8"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.550000011920929 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.007980599999427795 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.15338543057441711 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.26508671045303345 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.39017853140830994 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.019098035991191864 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.37892061471939087 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.1537572592496872 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5691391229629517 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 +"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.75 6 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.977656364440918 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 particle/accuracy_2_of_6‚1 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1442434936761856 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.13748739659786224 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.18773946166038513 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9873205423355103 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.4614197313785553 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5374014973640442 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4021349251270294 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.37621235847473145 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.30944058299064636 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.18837787210941315 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.2953064739704132 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.27327582240104675 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 +jet/accuracy_3_of_5‚0.08455580472946167 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 +"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.25 jet/accuracy_7_of_7‚0 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.2083333283662796 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5287730097770691 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 8"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.601190447807312 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 particle/accuracy_3_of_3‚1 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.991875410079956 ,jet/accuracy_5_of_5‚0.001792114693671465 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9270833134651184 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 :"Purity/*FRt2SRqqt1FBt/event_purity‚0.01309523917734623 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.007056144531816244 particle/accuracy_3_of_4‚1 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9994331002235413 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.013139831833541393 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_7‚1 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.08916987478733063 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.21867816150188446 *jet/accuracy_1_of_3‚0.8185186982154846 particle/accuracy_4_of_4‚1 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9824561476707458 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4879385828971863 +particle/specificity‚0.9477013349533081 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 *jet/accuracy_2_of_6‚0.4551282227039337 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.3083333373069763 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4285714328289032 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9464285969734192 particle/accuracy_1_of_7‚1 _step‚296 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5207858681678772 8"Purity/0FRt0SRqqt2FBt/event_purity‚0.336351215839386 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0022675737272948027 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.988946259021759 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.2520270347595215 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 +jet/accuracy_4_of_6‚0.03846153989434242 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6363846659660339 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.4095490276813507 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.16531895101070404 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.4382183849811554 +"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.75 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.41215068101882935 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.22451160848140717 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.09751074016094208 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.2787524163722992 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2675339877605438 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.3333333134651184 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.42356598377227783 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.3933449685573578 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1666666716337204 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.932539701461792 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.21996353566646576 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9904747605323792 particle/accuracy_1_of_2‚1 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1805555671453476 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9908382296562195 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 particle/accuracy_3_of_7‚1 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 6validation_average_jet_accuracy‚0.4009475111961365 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.41166412830352783 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3231481611728668 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 9"Purity/0FRt2SRqqt2FBt/event_purity‚0.0714285746216774 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_6‚1 5 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.90666663646698 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.07886485010385513 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.4295981824398041 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.4204545319080353 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 *jet/accuracy_1_of_1‚0.4398961067199707 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 % _timestamp‚1.7885269391941707e+09 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.4329014718532562 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_2_of_7‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8888888955116272 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 particle/accuracy_7_of_7‚0.5 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.2083333283662796 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5606567859649658 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 jet/accuracy_5_of_6‚0 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9184398055076599 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.49008622765541077 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.19473683834075928 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9735631346702576 ;"Purity/2FRt1SRqqt2FBt/event_purity‚0.022727273404598236 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2172619104385376 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.2857142984867096 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 ;"Purity/1FRt2SRqqt1FBt/event_purity‚0.008620689623057842 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.12719297409057617 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.12627504765987396 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4166666865348816 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.23333333432674408 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9875340461730957 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.34375 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_3_of_6‚1 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.45313534140586853 +jet/accuracy_3_of_6‚0.10897436738014221 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 jet/accuracy_6_of_6‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.4285714328289032 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2597792446613312 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7878787517547607 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 particle/accuracy_1_of_1‚1 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2222222238779068 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.3641975522041321 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.19032208621501923 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9812804460525513 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 jet/accuracy_6_of_7‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.37979573011398315 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9728301763534546 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9579861164093018 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.16222135722637177 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.1160871759057045 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.15226337313652039 *jet/accuracy_2_of_4‚0.3333214521408081 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.14768347144126892 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1544138789176941 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 particle/accuracy_2_of_5‚1 ;"Purity/3FRt*SRqqt1FBt/event_purity‚0.008333333767950535 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1639704555273056 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.17526724934577942 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ;"Purity/2FRt0SRqqt1FBt/event_purity‚0.012648810632526875 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 particle/accuracy_1_of_5‚1 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.31429171562194824 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.3844049870967865 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_4_of_5‚1 8"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.675438642501831 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.492946982383728 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3670520484447479 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.43030044436454773 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4206608235836029 *jet/accuracy_1_of_6‚0.9807692170143127 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3390637934207916 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 9"Purity/0FRt1SRqqt3FBt/event_purity‚0.1428571492433548 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.32914531230926514 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9842999577522278 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 <"Purity/3FRt*SRqqt*FBt/event_purity‚0.0023419205099344254 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9994331002235413 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.2308429479598999 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.011185983195900917 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.1197916641831398 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_5_of_6‚1 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5833333134651184 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.4719374477863312 *jet/accuracy_1_of_4‚0.8761470317840576 ,jet/accuracy_4_of_5‚0.017249103635549545 particle/accuracy_1_of_3‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4163374602794647 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.035467538982629776 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.00394606776535511 ;"Purity/3FRt0SRqqt*FBt/event_purity‚0.011111111380159855 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.8500000238418579 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.14086180925369263 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9826807379722595 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.17272444069385529 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7727272510528564 ,particle/sensitivity‚0.28416207432746887 8"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.503333330154419 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.3927299380302429 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0020502645056694746 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9563491940498352 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4024397134780884 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.3933449685573578 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.22680792212486267 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.22145061194896698 particle/accuracy_6_of_6‚1 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.32941171526908875 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.2083333432674408 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5917253494262695 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 & Purity/3FRt0SRqqt2FBt/FRt_purity‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1626984179019928 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.4538644850254059 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.27444660663604736 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3716356158256531 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 *jet/accuracy_3_of_4‚0.0628167986869812 (particle/accuracy‚0.8440755009651184 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.3499999940395355 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.18098396062850952 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5074579119682312 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5815476179122925 +"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.25 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 9"Purity/*FRt1SRqqt3FBt/event_purity‚0.1111111119389534 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.07894736528396606 particle/accuracy_2_of_2‚1 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 :"Purity/2FRt*SRqqt2FBt/event_purity‚0.05426356941461563 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ,jet/accuracy_4_of_4‚0.008785532787442207 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.24166668951511383 :"Purity/*FRt2SRqqt2FBt/event_purity‚0.03999999910593033 jet/accuracy_1_of_7‚1 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9966569542884827 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 6 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.903508722782135 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5134441256523132 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_5_of_7‚1 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6723325252532959 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 jet/accuracy_3_of_7‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.3294612765312195 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.013594469986855984 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4254404902458191 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.42527228593826294 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 *jet/accuracy_2_of_3‚0.2203972488641739 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.3951389491558075 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.25925925374031067 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9967953562736511 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.4056878983974457 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9990079402923584 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.46910926699638367 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9947887659072876 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.561768114566803 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9985569715499878 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.4757598042488098 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5708333253860474 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.020181190222501755 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5370370745658875 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 *jet/accuracy_1_of_5‚0.9336532950401306 *jet/accuracy_2_of_5‚0.4460829198360443 particle/accuracy_2_of_3‚1 particle/accuracy_2_of_4‚1 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5882353186607361 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5684332251548767 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.19879211485385895 ."Purity/3FRt0SRqqt1FBt/event_purity‚0.03125 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.8666666746139526 particle/accuracy_4_of_7‚1 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2253185212612152 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2069786936044693 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.31450676918029785 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.22640535235404968 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9879111051559448 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.27097058296203613 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.01867816224694252 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.3333333134651184 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492¨—Ñ6Ïv¦‚(:ë Ý‚ëÔ‚þù' disk.nvme0n1.out‚116909.90087890625 disk.sda1.out‚0 disk.sda3.out‚ 177.59375$ memory_percent‚99.96185302734375 proc.memory.availableMB‚6.25 proc.cpu.threads‚17 network.sent‚ 9.336583e+06 network.recv‚ 1.981173e+06) disk./.usagePercent‚78.99330152190541$ disk.nvme0c0n1.in‚14521.58203125) disk.nvme0c0n1.out‚116909.90087890625 disk.sda1.in‚0 disk.drbd1260.out‚ 0.8515625 disk.sda2.in‚0 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda.out‚ 177.59375! proc.memory.rssMB‚ 3091.171875* proc.memory.percent‚18.867015838623047 cpu‚25.404327966495764% disk./.usageGB‚1177.0572776794434" disk.nvme0n1.in‚14521.58203125 disk.dm-1.out‚ 177.59375 disk.sda2.out‚0 disk.dm-0.in‚14521.58203125$ disk.dm-0.out‚116909.90087890625 disk.sda.in‚0 disk.sda3.in‚0 5Ô䧂(:„ Ý‚ëÔ§Æçù gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚92.744' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.5488 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0æSDM‚¨‚(:÷ ì‚ëÔ¹µ¨ù disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.85546875( proc.memory.availableMB‚ 350.33984375% disk./.usageGB‚1176.9311447143555$ disk.nvme0c0n1.in‚14521.58203125% disk.nvme0n1.out‚117309.689453125 disk.sda2.out‚0" proc.memory.rssMB‚ 2917.5859375* proc.memory.percent‚17.807531356811523 cpu‚25.40412288249365" disk.dm-0.out‚117309.689453125 disk.dm-1.in‚0 disk.sda2.in‚0$ memory_percent‚97.86169528961182 proc.cpu.threads‚17 network.sent‚ 9.461258e+06 network.recv‚ 1.987788e+06) disk./.usagePercent‚78.98483663278574 disk.sda.in‚0 disk.sda.out‚ 178.03515625 disk.sda3.in‚0 disk.sda3.out‚ 178.03515625' disk.nvme0c0n1.out‚117309.689453125 disk.dm-0.in‚14521.58203125" disk.nvme0n1.in‚14521.58203125 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 178.03515625 íPHf©‚(:… ì‚ëÔ¸ÂÙú gpu.0.gpu‚19.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚92.813' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.5626 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrorÊVÆR*s‚0# gpu.0.uncorrectedMemoryErrors‚0JO ™‘ª‚(:† û‚ëÔœ†Šù gpu.0.gpu‚23.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚103.722' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.7444 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0äû›«‚(:ô û‚ëÔÕßüú disk.sda2.out‚0 disk.dm-0.in‚14521.58203125" disk.dm-0.out‚117664.783203125 disk.sda.out‚ 178.4765625 disk.sda3.out‚ 178.4765625' disk.nvme0c0n1.out‚117664.783203125 network.sent‚ 9.466608e+06 disk.sda.in‚0" disk.nvme0n1.in‚14521.58203125 disk.sda1.in‚0 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.85546875# memory_percent‚97.8618860244751) disk./.usagePercent‚79.01839461238903% disk./.usageGB‚1177.4311828613281 disk.sda3.in‚0$ disk.nvme0c0n1.in‚14521.58203125 disk.dm-1.out‚ 178.4765625 disk.drbd1260.in‚0( proc.memory.availableMB‚ 350.30859375 proc.cpu.threads‚17% disk.nvme0n1.out‚117664.783203125 disk.sda1.out‚0" proc.memory.rssMB‚ 2917.5859375* proc.memory.percent‚17.807531356811523 cpu‚25.403819567172825 network.recv‚ 1.991141e+06 disk.sda2.in‚0/–Æf2¬- _wandbruntime‚6219  _runtime‚6219qW˜Ç­Á & lr-AdamW/pg2‚0.0008095094842020421 trainer/global_step‚3649 % _timestamp‚1.7885269808467913e+09 _runtime‚6219.067597779 _step‚297 & lr-AdamW/pg1‚0.0008095094842020421¼AÌ®Æ & lr-AdamW/pg2‚0.0008095094842020421 trainer/global_step‚3649 % _timestamp‚1.7885269808467913e+09 _runtime‚6219.067597779 _step‚297 & lr-AdamW/pg1‚0.0008095094842020421©[á_,2¯- _wandbruntime‚6219  _runtime‚6219¥7ëqÜ'°ÖO ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚149094199261184 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚41643918 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚664 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚666 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚157901961566720 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 299008000 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚141830167319888 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚69623437 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7258284742656 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚99374079 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚165285641603072 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚99347735 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚156352466000896 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 168997518 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚6219.070410401 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚156352717672448 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚69623437 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚99374079 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚149094181258240 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 trainer/global_step‚3649 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚156352466000896 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 168971172 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚99347735 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚157901985506816 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7383656096256 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚149068118309904 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26449578 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360ìJ8á'±ÛO trainer/global_step‚3649 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚156352466000896 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 168971172 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚99347735 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚157901985506816 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7383656096256 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚149068118309904 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26449578 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚149094199261184 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚41643918 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚664 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚666 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚157901961566720 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 299008000 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚141830167319888 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚69623437 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7258284742656 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚99374079 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚165285641603072 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚99347735 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚156352466000896 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 168997518 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚6219.070410401 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚156352717672448 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚69623437 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚99374079 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚149094181258240 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319ªFŸ 2²- _wandbruntime‚6221  _runtime‚6221GI»\å&³ßM GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 _step‚299 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41654720 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚69641714 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚141867714716672 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68110588 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚674 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 308254720 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 trainer/global_step‚3649 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41654718 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚676 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26456544 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚165329000298496 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7239892439850 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚69641714 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚149107607156522 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚99400797 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚99400797 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 169042511 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 169042511 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚69641712 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚156393716063744 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚149133467295232 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚149133485298176 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7260482450432 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚156393967748608 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚149133467295232 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7385354761728 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 169016139 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚149133485298176 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚68111264 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚99374427 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚149107361135870 % _timestamp‚1.7885269829532006e+09 _runtime‚ 6221.17806082 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7260248768512 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7260248768512 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚141867697677312 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26455870 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚165329308553216 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚99374427 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 169016139 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7260482450432 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319y«Ì§´äM T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚156393967748608 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_en)(iDC%d/active.all.peak‚32249 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚149133467295232 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7385354761728 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 169016139 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚149133485298176 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚68111264 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚99374427 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚149107361135870 % _timestamp‚1.7885269829532006e+09 _runtime‚ 6221.17806082 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7260248768512 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7260248768512 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚141867697677312 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26455870 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚165329308553216 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚99374427 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 169016139 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7260482450432 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 _step‚299 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41654720 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚69641714 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚141867714716672 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68110588 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚674 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 308254720 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 trainer/global_step‚3649 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41654718 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚676 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26456544 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚165329000298496 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7239892439850 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚69641714 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚149107607156522 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚99400797 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚99400797 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 169042511 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 169042511 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚69641712 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚156393716063744 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚149133467295232 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚149133485298176 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7260482450432 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚156393967748608 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 PDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚158414983557120 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚167469478340096 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42184018 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚158415235228672 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚699 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚158415235228672 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚151058501110784 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚701 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚159986184009216 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7483608890368 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 100685687 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7335643010080 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42184016 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 314559488 _runtime‚6303.289192867 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 100712031 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26797976 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚151058501110784 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7356500449280 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 % _timestamp‚1.7885270650694168e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7356500449280 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚159986160069120 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7335871991164 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚70537189 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7356734117888 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚151058483107840 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚68981293 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚151058483107840 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚143706685191088 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 171222874ØèTÉßO M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 100685687 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7335643010080 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42184016 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 314559488 _runtime‚6303.289192867 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 100712031 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26797976 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚151058501110784 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7356500449280 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 % _timestamp‚1.7885270650694168e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7356500449280 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 H>DeviceStatsMonitor.on_train_batch_start/allð×RœÌ ocation.all.current‚26346 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚159986160069120 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7335871991164 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚70537189 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7356734117888 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚151058483107840 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚68981293 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚151058483107840 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚143706685191088 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 171222874 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚167469792899584 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚151042328201168 trainer/global_step‚3699 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26797277 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚158414983557120 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚167469478340096 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42184018 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚158415235228672 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚699 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚158415235228672 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚151058501110784 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚701 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚159986184009216 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7483608890368 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0®s{]N2Ê- _wandbruntime‚6304  _runtime‚6304jƒÈaê&ËäM WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚151097769144832 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 320837632 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚143744232587872 _runtime‚6304.771126431 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 171294213 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7358464475136 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚695 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚70555462 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68999412 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 100738749 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚160027844039168 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 171294213 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 296897536 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 171267841 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7358464475136 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚70555462 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42194818 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26804594 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚143744215548512 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚151097787147776 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7358698157056 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚70555464 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚151097787147776 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚693 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 % _timestamp‚1.7885270665476725e+09 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚158456485304832 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚158456485304832 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚69000107 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚160027867979264 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 trainer/global_step‚3699 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚303 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 100712379 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7485602108416 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7337830480358 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 100712379 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚167513149250048 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68999412 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 100738749 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚160027844039168 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 171294213 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 296897536 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 171267841 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7358464475136 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚70555462 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42194818 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26804594 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚143744215548512 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚151097787147776 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7358698157056 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚70555464 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚151097787147776 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚693 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 % _timestamp‚1.7885270665476725e+09 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚158456485304832 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚158456485304832 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚69000107 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚160027867979264 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 trainer/global_step‚3699 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚303 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 100712379 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7485602108416 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7337830480358 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 100712379 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚167513149250048 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0¯ˆº7T2Í- _wandbruntime‚6304  _runtime‚6304­Š­ó^ÎØ 2loss/SRqqt2/assignment_loss‚1.5906564563512802 /loss/FRt1/detection_loss‚2.6998410791885563 .loss/FRt2/detection_loss‚2.969891936690719 /loss/FRt3/detection_loss‚1.8490306954564792 %loss/total_loss‚44.91795182098234 0loss/FRt3/assignment_loss‚1.8112992314355714 1loss/FBt1/assignment_loss‚0.24054975009682586 1loss/SRqqt3/detection_loss‚1.1512522565845018 epoch‚3 % _timestamp‚1.7885270665570438e+09 1loss/FBt4/assignment_loss‚0.09598004991547116 _step‚304 0loss/FRt1/assignment_loss‚3.1697953605070346 /loss/FRt2/assignment_loss‚4.263830602169037 0loss/SRqqt1/detection_loss‚1.672989849672943 0loss/SRqqt2/detection_loss‚2.443693790265492 .loss/FBt1/detection_loss‚2.765023106878454 /loss/FRt4/detection_loss‚1.9178762056843632 1loss/SRqqt4/detection_loss‚2.5983488427268133 2loss/FBt3/assignment_loss‚0.062357227559550664 2loss/SRqqt4/assignment_loss‚2.1661282148626118 /loss/FBt2/detection_loss‚2.3214791297912596 2loss/SRqqt3/assignment_loss‚1.1816235114341584 .loss/FBt4/detection_loss‚0.648425416922883 trainer/global_step‚3699 /loss/FBt3/detection_loss‚2.3824864506721495 0loss/FRt4/assignment_loss‚2.9337720903483304 2loss/SRqqt1/assignment_loss‚1.7469734196535875 1loss/FBt2/assignment_loss‚0.23464714611427315 _runtime‚6304.778222316O~š(cÏÝ 0loss/FRt1/assignment_loss‚3.1697953605070346 /loss/FRt2/assignment_loss‚4.263830602169037 0loss/SRqqt1/detection_loss‚1.672989849672943 0loss/SRqqt2/detection_loss‚2.443693790265492 .loss/FBt1/detection_loss‚2.765023106878454 /loss/FRt4/detection_loss‚1.9178762056843632 1loss/SRqqt4/detection_loss‚2.5983488427268133 2loss/FBt3/assignment_loss‚0.062357227559550664 2loss/SRqqt4/assignment_loss‚2.1661282148626118 /loss/FBt2/detection_loss‚2.3214791297912596 2loss/SRqqt3/assignment_loss‚1.1816235114341584 .loss/FBt4/detection_loss‚0.648425416922883 trainer/global_step‚3699 /loss/FBt3/detection_loss‚2.3824864506721495 0loss/FRt4/assignment_loss‚2.9337720903483304 2loss/SRqqt1/assignment_loss‚1.7469734196535875 1loss/FBt2/assignment_loss‚0.23464714611427315 _runtime‚6304.778222316 2loss/SRqqt2/assignment_loss‚1.5906564563512802 /loss/FRt1/detection_loss‚2.6998410791885563 .loss/FRt2/detection_loss‚2.969891936690719 /loss/FRt3/detection_loss‚1.8490306954564792 %loss/total_loss‚44.91795182098234 0loss/FRt3/assignment_loss‚1.8112992314355714 1loss/FBt1/assignment_loss‚0.24054975009682586 1loss/SRqqt3/detection_loss‚1.1512522565845018 epoch‚3 % _timestamp‚1.7885270665570438e+09 1loss/FBt4/assignment_loss‚0.09598004991547116 _step‚304°LZ|¾‘Ђ(:† äƒëÔëÐØ÷ gpu.0.gpu‚29.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚108.991' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7982 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚01Ó˜ítÑ‚(:é äƒëÔ¸á»ø cpu‚25.401617429848102) disk./.usagePercent‚79.01671750703223$ disk./.usageGB‚1177.406192779541 disk.sda1.out‚0 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 182.03515625 disk.nvme0n1.in‚14838.25 network.sent‚ 9.608418e+06 network.recv‚ 2.022301e+06$ disk.dm-0.out‚119992.27978515625 disk.nvme0c0n1.in‚14838.25) disk.nvme0c0n1.out‚119992.27978515625' disk.nvme0n1.out‚119992.27978515625# memory_percent‚97.8708028793335( proc.memory.availableMB‚ 348.84765625 disk.drbd1260.out‚ 0.8671875 disk.sda.in‚0* proc.memory.percent‚17.807531356811523 proc.cpu.threads‚17 disk.sda1.in‚0 disk.dm-1.out‚ 182.03515625 disk.sda2.in‚0 disk.dm-0.in‚14838.25 disk.sda.out‚ 182.03515625 disk.sda3.in‚0" proc.memory.rssMB‚ 2917.5859375dô’ëÒ‚(:„ óƒëÔ‰æÐö gpu.0.gpu‚6.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚92.813' gpu.0.enforcedPowerLimitWattsªuâh¤‚500.0 gpu.0.powerPercent‚18.5626 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆàoovÓ‚(:ë óƒëÔÜ÷¢ø cpu‚25.401341057812658 proc.cpu.threads‚17) disk./.usagePercent‚79.01829246538287$ disk.dm-0.out‚120391.00634765625 disk.sda3.in‚0 disk.sda3.out‚ 182.44140625' disk.nvme0n1.out‚120391.00634765625 disk.dm-1.out‚ 182.44140625 disk.drbd1260.in‚0 disk.sda2.out‚0 network.sent‚ 9.71616e+06 disk.sda.in‚0 disk.nvme0n1.in‚14838.25 disk.sda2.in‚0$ memory_percent‚97.87099361419678( proc.memory.availableMB‚ 348.81640625" proc.memory.rssMB‚ 2917.5859375* proc.memory.percent‚17.807531356811523 disk.dm-0.in‚14838.25 disk.sda.out‚ 182.44140625 disk.sda1.in‚0 disk.dm-1.in‚0 network.recv‚ 2.027674e+06% disk./.usageGB‚1177.4296607971191 disk.nvme0c0n1.in‚14838.25) disk.nvme0c0n1.out‚120391.00634765625 disk.sda1.out‚0 disk.drbd1260.out‚ 0.87109375¶*Šn}Ô‚(:ò ‚„ëÔªÜÔ÷ disk.sda3.out‚ 183.15625$ memory_percent‚97.87273406982422 network.sent‚ 9.721291e+06) disk./.usagePercent‚79.01829246538287" disk.nvme0n1.in‚15030.26953125 disk.dm-1.in‚0 disk.dm-1.out‚ 183.15625 disk.dm-0.in‚15030.26953125 disk.sda.out‚ 183.15625 disk.sda3.in‚0$ disk.nvme0c0n1.in‚15030.26953125 disk.drbd1260.in‚0$ disk.dm-0.out‚120766.34619140625% proc.memory.availableMB‚ 348.53125" proc.memory.rssMB‚ 2917.5859375 cpu‚25.401108469551385 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda.in‚0* proc.memory.percent‚17.807531356811523 network.recv‚ 2.031432e+06% disk./.usageGB‚1177.4296607971191) disk.nvme0c0n1.out‚120766.34619140625' disk.nvme0n1.out‚120766.34619140625 disk.sda1.in‚0 disk.drbd1260.out‚ 0.87109375ã½±›Õ‚(: ‚„ëÔù÷ˆø gpu.0.gpu‚10.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚99.207' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚19.841399999999997 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0)^…œÖ‚(:‘ ‘„ëÔ¥úÈú gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚136.178' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.235599999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0óv&Ä|ׂ(:ñ ‘„ëÔÆÊÙþ network.recv‚ 2.035541e+06) disk./.usagePercent‚79.01829246538287% disk./.usageGB‚1177.4296607971191 disk.drbd1260.out‚0.875 disk.dm-0.in‚15030.26953125) disk.nvme0c0n1.out‚120824.48681640625 disk.dm-1.out‚ 183.7578125 disk.sda2.in‚0 disk.sda3.out‚ 183.7578125$ disk.nvme0c0n1.in‚15030.26953125 disk.sda1.in‚0 disk.dm-1.in‚0$ memory_percent‚97.86109924316406 cpu‚25.40081415663739 disk.drbd1260.in‚0 disk.sda2.out‚0 disk.sda3.in‚0" disk.nvme0n1.in‚15030.26953125' disk.nvme0n1.out‚120824.48681640625 disk.sda1.out‚0$ proc.memory.availableMB‚350.4375 network.sent‚ 9.729061e+06$ disk.dm-0.out‚120824.48681640625 disk.sda.in‚0 disk.sda.out‚ 183.7578125" proc.memory.rssMB‚ 2917.5859375* proc.memory.percent‚17.807531356811523 proc.cpu.threads‚17šƒíØ‚(:‚  „ëÔëÝÎ÷ gpu.0.gpu‚42.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚113.4' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.68 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0μJxÙ‚(:í  „ëÔù„žû network.sent‚ 9.735263e+06$ disk./.usageGB‚1176.344009399414 disk.dm-1.out‚ 184.40625 disk.drbd1260.in‚0 disk.drbd1260.out‚0.875 disk.sda.in‚0) disk.nvme0c0n1.out‚121291.31103515625' disk.nvme0n1.out‚121291.31103515625) disk./.usagePercent‚78.94543348916068 disk.sda2.out‚0" disk.nvme0n1.in‚15030.26953125 disk.sda1.out‚0# memory_percent‚97.8626012802124( proc.memory.availableMB‚ 350.19140625" proc.memory.rssMB‚ 2917.5859375* proc.memory.percent‚17.807531356811523 network.recv‚ 2.040307e+06 disk.dm-1.in‚0 disk.dm-0.in‚15030.26953125$ disk.dm-0.out‚121291.31103515625 disk.sda.out‚ 184.40625 disk.sda3.in‚0 disk.sda3.out‚ 184.40625$ disk.nvme0c0n1.in‚15030.26953125 disk.sda2.in‚0 disk.sda1.in‚0 cpu‚25.40017918348745 proc.cpu.threads‚17QÆÄ*2Ú- _wandbruntime‚6378  _runtime‚6378PõUñÇÛÁ % _timestamp‚1.7885271405389378e+09 _runtime‚6378.758529156 _step‚305 & lr-AdamW/pg1‚0.0008007737651508242 & lr-AdamW/pg2‚0.0008007737651508242 trainer/global_step‚3749bS¼ÌÜÆ trainer/global_step‚3749 % _timestamp‚1.7885271405389378e+09 _runtime‚6378.758529156 _step‚305 & lr-AdamW/pg1‚0.0008007737651508242 & lr-AdamW/pg2‚0.0008007737651508242±ª®02Ý- _wandbruntime‚6378  _runtime‚6378yiB˜à'ÞÚO XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27150028 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚306 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚160477501113344 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 173500922 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚69874146 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 102049983 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 102023639 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚69873458 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚153017030133320 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚153022802960384 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 trainer/global_step‚3749 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚71450939 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7454949824512 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚686 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚145583203062288 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7433810031672 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚160477501113344 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7583156397568 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚153022802960384 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7454716155904 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 296910848 % _timestamp‚1.7885271405400736e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 320850944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚160477752784896 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 102049983 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42724116 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚688 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚6378.759748734 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚160477752784896 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 102023639 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚71450937 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532›ØÀ€å'ßßO ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42724116 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚688 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚6378.759748734 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚160477752784896 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 102023639 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚71450937 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27150028 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚306 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚160477501113344 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 173500922 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚69874146 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 102049983 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 102023639 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚69873458 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚153017030133320 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚153022802960384 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 trainer/global_step‚3749 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚71450939 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7454949824512 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚686 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚145583203062288 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7433810031672 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚160477501113344 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7583156397568 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚153022802960384 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7454716155904 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 296910848 % _timestamp‚1.7885271405400736e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 320850944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚160477752784896 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 102049983 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309²>š3.2à- _wandbruntime‚6379  _runtime‚6379<*ˆkSáäM I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚162112042541568 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚6379.690709232 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27156423 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚153062070994432 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚160519002861056 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 173545915 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 _step‚307 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 102050331 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 102050331 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚153056518979938 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚671 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7456913863680 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 % _timestamp‚1.7885271414703681e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 173545915 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚71469212 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7585146902016 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7435539539574 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7456680181760 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7585443799552 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚673 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚160518751176192 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚153062088997376 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 320837632 E?DeviceStatsMonitor.on_train_batch_end/oversiz®%OÜ— e_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7435768520866 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69891341 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7456913863680 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 102076701 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 173519543 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69891341 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7456913863680 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 102076701 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 173519543 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚6379.690709232 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27156423 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚153062070994432 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚160519002861056 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 173545915 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 _step‚307 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 102050331 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 102050331 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚153056518979938 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚671 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7456913863680 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 % _timestamp‚1.7885271414703681e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 173545915 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚71469212 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7585146902016 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7435539539574 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7456680181760 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7585443799552 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚673 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚160518751176192 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚153062088997376 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 320837632³Jíªd2ã- _wandbruntime‚6379  _runtime‚6379-…A\äÖ 2loss/SRqqt1/assignment_loss‚1.5127881518673412 2loss/SRqqt3/assignment_loss‚1.2350816100772302 1loss/SRqqt1/detection_loss‚1.8987033318641573 0loss/SRqqt3/detection_loss‚0.947952815009789 /loss/FBt4/detection_loss‚0.6193085871636868 _runtime‚6379.692394749 trainer/global_step‚3749 % _timestamp‚1.7885271414730186e+09 2loss/SRqqt2/assignment_loss‚1.9975024940712112 1loss/FBt2/assignment_loss‚0.15590943403892904 1loss/FBt3/assignment_loss‚0.05855196843620953 /loss/FRt3/detection_loss‚1.7565875111476463 .loss/FRt4/detection_loss‚2.006418898852693 1loss/SRqqt2/detection_loss‚2.6835586556366513 0loss/FRt1/assignment_loss‚3.6673474311828613 1loss/SRqqt4/assignment_loss‚2.142233619896265 /loss/FRt2/detection_loss‚3.2330421711269177 1loss/SRqqt4/detection_loss‚2.7587755780953627 /loss/FBt3/detection_loss‚3.1216375402041843 0loss/FRt3/assignment_loss‚2.0669833079856983 %loss/total_loss‚47.31666142063844 /loss/FRt2/assignment_loss‚4.097246166906859 0loss/FRt4/assignment_loss‚2.7233284889040767 .loss/FRt1/detection_loss‚2.931396492065922 0loss/FBt1/assignment_loss‚0.1764089043494592 1loss/FBt4/assignment_loss‚0.13262550061266687 epoch‚3 _step‚308 .loss/FBt1/detection_loss‚2.983299417929216 /loss/FBt2/detection_loss‚2.4099733432133994%î“aåÛ /loss/FRt2/assignment_loss‚4.097246166906859 0loss/FRt4/assignment_loss‚2.7233284889040767 .loss/FRt1/detection_loss‚2.931396492065922 0loss/FBt1/assignment_loss‚0.1764089043494592 1loss/FBt4/assignment_loss‚0.13262550061266687 epoch‚3 _step‚308 .loss/FBt1/detection_loss‚2.983299417929216 /loss/FBt2/detection_loss‚2.4099733432133994 2loss/SRqqt1/assignment_loss‚1.5127881518673412 2loss/SRqqt3/assignment_loss‚1.2350816100772302 1loss/SRqqt1/detection_loss‚1.8987033318641573 0loss/SRqqt3/detection_loss‚0.947952815009789 /loss/FBt4/detection_loss‚0.6193085871636868 _runtime‚6379.692394749 trainer/global_step‚3749 % _timestamp‚1.7885271414730186e+09 2loss/SRqqt2/assignment_loss‚1.9975024940712112 1loss/FBt2/assignment_loss‚0.15590943403892904 1loss/FBt3/assignment_loss‚0.05855196843620953 /loss/FRt3/detection_loss‚1.7565875111476463 .loss/FRt4/detection_loss‚2.006418898852693 1loss/SRqqt2/detection_loss‚2.6835586556366513 0loss/FRt1/assignment_loss‚3.6673474311828613 1loss/SRqqt4/assignment_loss‚2.142233619896265 /loss/FRt2/detection_loss‚3.2330421711269177 1loss/SRqqt4/detection_loss‚2.7587755780953627 /loss/FBt3/detection_loss‚3.1216375402041843 0loss/FRt3/assignment_loss‚2.0669833079856983 %loss/total_loss‚47.31666142063844´ØûHp‘æ‚(:† ¯„ëÔù÷êö gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚33.0 gpu.0.powerWatts‚113.934' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.7868 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Êò_…ç‚(:ú ¯„ëÔÒßÃù" disk.nvme0n1.in‚15030.26953125' disk.nvme0n1.out‚121628.28955078125$ disk.dm-0.out‚121628.28955078125 disk.sda.in‚0 disk.sda.out‚ 184.8359375" proc.memory.rssMB‚ 2917.5859375 proc.cpu.threads‚17 network.sent‚ 9.738425e+06) disk./.usagePercent‚78.94700844751132% disk./.usageGB‚1176.3674774169922 disk.sda3.out‚ 184.8359375 disk.dm-1.out‚ 184.8359375 cpu‚25.399416486412385 disk.sda3.in‚0 disk.sda1.out‚0 disk.drbd1260.in‚0$ memory_percent‚97.86419868469238' proc.memory.availableMB‚ 349.9296875* proc.memory.percent‚17.807531356811523$ disk.nvme0c0n1.in‚15030.26953125) disk.nvme0c0n1.out‚121628.28955078125 disk.sda1.in‚0 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.87890625 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚15030.26953125 network.recv‚ 2.043916e+06` .é‘è‚(:† ¾„ëÔÅòãö gpu.0.gpu‚24.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚108.712' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7424 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚02#ué‚(:ô ¾„ëÔ•Ÿþø( disk./.usagePercent‚78.9470112635942" disk.nvme0c0n1.in‚ 15052.109375 disk.sda1.in‚0 disk.dm-1.in‚0 disk.sda2.out‚0$ disk.dm-0.out‚121951.64892578125 disk.sda3.out‚ 185.54296875 cpu‚25.398763989641438 network.sent‚ 9.84624e+06$ disk./.usageGB‚1176.367519378662) disk.nvme0c0n1.out‚121951.64892578125 disk.sda1.out‚0 disk.dm-1.out‚ 185.54296875 disk.drbd1260.in‚0 disk.sda.in‚0$ memory_percent‚97.86267280578613 disk.nvme0n1.in‚ 15052.109375 disk.drbd1260.out‚ 0.87890625 disk.sda2.in‚0 disk.dm-0.in‚ 15052.109375 disk.sda3.in‚0' proc.memory.availableMB‚ 350.1796875" proc.memory.rssMB‚ 2917.5859375* proc.memory.percent‚17.807531356811523 network.recv‚ 2.049428e+06' disk.nvme0n1.out‚121951.64892578125 disk.sda.out‚ 185.54296875 proc.cpu.threads‚17E¸žœê‚(:‘ Í„ëÔýç…÷ gpu.0.gpu‚23.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚103.479' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚20.695800000000002 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0bäx ‡ë‚(:ü Í„ëÔ„Ä™ø disk.dm-0.in‚15218.80859375$ disk.dm-0.out‚122192.49267578125 disk.sda.in‚0 disk.sda3.out‚ 186.05859375$ memory_percent‚97.86124229431152' proc.memory.availableMB‚ 350.4140625 disk.dm-1.out‚ 186.05859375 disk.drbd1260.out‚ 0.8828125% disk./.usageGB‚1176.3674507141113$ disk.nvme0c0n1.in‚15218.80859375 disk.sda3.in‚0 cpu‚25.398155139171017 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda2.out‚0" proc.memory.rssMB‚ 2917.5859375* proc.memory.percent‚17.807531356811523 proc.cpu.threads‚17 network.sent‚ 9.852305e+06) disk./.usagePercent‚78.94700665545858) disk.nvme0c0n1.out‚122192.49267578125" disk.nvme0n1.in‚15218.80859375' disk.nvme0n1.out‚122192.49267578125 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda.out‚ 186.05859375 network.recv‚ 2.053703e+06 disk.sda1.in‚0Tÿì‚(:„ Ü„ëÔÖØ¡ö gpu.0.gpu‚17.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚98.675' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.735 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0·›Oþ†í‚(:û Ü„ëÔÕ ëü disk.dm-0.in‚15218.80859375* proc.memory.percent‚17.807531356811523$ disk.dm-0.out‚122283.38330078125 disk.sda.out‚ 186.50390625 disk.sda3.out‚ 186.50390625$ memory_percent‚97.86276817321777' proc.memory.availableMB‚ 350.1640625" proc.memory.rssMB‚ 2917.5859375 proc.cpu.threads‚17 network.recv‚ 2.058107e+06% disk./.usageGB‚1176.3674507141113 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.8828125 disk.sda2.out‚0 disk.sda3.in‚0' disk.nvme0n1.out‚122283.38330078125 disk.sda1.out‚0 disk.sda.in‚0$ disk.nvme0c0n1.in‚15218.80859375) disk.nvme0c0n1.out‚122283.38330078125" disk.nvme0n1.in‚15218.80859375 disk.sda1.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 186.50390625 cpu‚25.39804482675029 network.sent‚ 9.860431e+06) disk./.usagePercent‚78.94700665545858 disk.sda2.in‚0 ©ƒ2î- _wandbruntime‚6443  _runtime‚6443¶¥ïÅÅï¿ & lr-AdamW/pg1‚0.0007962987467483824 & lr-AdamW/pg2‚0.0007962987467483824 trainer/global_step‚3799 # _timestamp‚1.78852720538387e+09 _runtime‚6443.603346685 _step‚309B‚ÙÊðÄ # _timestamp‚1.78852720538387e+09 _runtime‚6443.603346685 _step‚309 & lr-AdamW/pg1‚0.0007962987467483824 & lr-AdamW/pg2‚0.0007962987467483824 trainer/global_step‚3799µ•Ù±¾2ñ- _wandbruntime‚6443  _runtime‚6443ˆ€êtß'òÙO ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚154987104809984 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7552931862528 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚72364689 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 103361591 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚164154581014016 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚171837820948480 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚154987104809984 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚693 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚72364687 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27497027 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚164154557073920 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚162540270341120 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚162540018669568 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 103387935 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚162540270341120 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚154987086807040 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚154987086807040 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚70760550 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 103387935 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 103361591 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7531748072180 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚147459737972848 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚147459720933488 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27496334 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 290619392 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 _runtime‚ 6443.60470181 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7553165531136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 % _timestamp‚1.7885272053849654e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 175726278 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7552931862528 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚72364687 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚695 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 314559488 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7682949315072 _step‚310 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7553165531136 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309ÇŸû?ÖóÞO ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7552931862528 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚72364687 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚695 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 314559488 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7682949315072 _step‚310 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7553165531136 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 50%JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚154987104809984 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7552931862528 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚72364689 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 103361591 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚164154581014016 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚171837820948480 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚154987104809984 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚693 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚72364687 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27497027 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚164154557073920 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚162540270341120 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚162540018669568 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 103387935 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚162540270341120 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚154987086807040 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚154987086807040 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚70760550 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 103387935 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 103361591 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7531748072180 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚147459737972848 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚147459720933488 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27496334 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 290619392 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 _runtime‚ 6443.60470181 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7553165531136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 % _timestamp‚1.7885272053849654e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 175726278 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0¶9s:2ô- _wandbruntime‚6444  _runtime‚6444eoö6ê&õäM ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 103414653 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27503046 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286411776 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚155030974891646 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚147497268330272 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7533706561374 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚171881183707136 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚162581520417280 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚70778746 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚43275018 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310351872 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27503726 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚155026372844032 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 175797617 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 175797617 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 trainer/global_step‚3799 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚162581520417280 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚171881494059008 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚164196241043968 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 103388283 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7555129570304 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚162581268732416 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚155026390846976 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7684942663168 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7554895888384 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7533477580082 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚72382962 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7685229074944 _runtime‚6444.661843459 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7555129570304 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 103414653 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚680 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 _step‚311 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 175771245 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 % _timestamp‚1.7885272064417922e+09 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚155026372844032 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚147497251290912 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70778064 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚72382962 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚155030728870994½¥Åï&öéM K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70778064 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚72382962 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚155030728870994 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 103414653 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27503046 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286411776 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚155030974891646 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚147497268330272 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7533706561374 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚171881183707136 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚162581520417280 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚70778746 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚43275018 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310351872 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27503726 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚155026372844032 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 175797617 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 175797617 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 trainer/global_step‚3799 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚162581520417280 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚171881494059008 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚164196241043968 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 103388283 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7555129570304 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚162581268732416 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚155026390846976 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7684942663168 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7554895888384 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7533477580082 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚72382962 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7685229074944 _runtime‚6444.661843459 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7555129570304 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 103414653 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚680 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 _step‚311 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 175771245 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 % _timestamp‚1.7885272064417922e+09 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚155026372844032 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚147497251290912 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104·M£T2÷- _wandbruntime‚6444  _runtime‚6444הNjZøÔ 0loss/FRt2/assignment_loss‚2.8230631223746707 2loss/SRqqt3/assignment_loss‚1.1523419902423175 1loss/SRqqt4/detection_loss‚2.6610685609005116 &loss/total_loss‚44.866846654060325 1loss/SRqqt4/assignment_loss‚1.986108954306002 0loss/FBt1/assignment_loss‚0.3567955703395207 /loss/FRt3/detection_loss‚1.5349855144054463 1loss/SRqqt2/detection_loss‚2.4261492558888027 /loss/FRt1/assignment_loss‚3.559944959481557 0loss/SRqqt3/detection_loss‚0.955813426192913 .loss/FBt2/detection_loss‚2.500784151694354 ,loss/FRt2/detection_loss‚2.9334745321955 /loss/FRt4/detection_loss‚2.0111504563531124 epoch‚3 trainer/global_step‚3799 0loss/FRt3/assignment_loss‚1.6332194369572859 0loss/FRt4/assignment_loss‚3.0153496279272924 0loss/FBt2/assignment_loss‚0.4104302875861688 1loss/FBt3/assignment_loss‚0.03380341596694052 1loss/SRqqt1/detection_loss‚1.9954872783273458 _runtime‚6444.663805106 2loss/SRqqt1/assignment_loss‚1.6195677617021527 1loss/FBt4/assignment_loss‚0.17679355035238953 /loss/FBt3/detection_loss‚3.3324320554733275 /loss/FBt4/detection_loss‚0.9610858009827714 2loss/SRqqt2/assignment_loss‚1.6590996887002671 /loss/FRt1/detection_loss‚2.9003520647684735 .loss/FBt1/detection_loss‚2.227545190941204 % _timestamp‚1.7885272064444852e+09 _step‚312–r_ùÙ _runtime‚6444.663805106 2loss/SRqqt1/assignment_loss‚1.6195677617021527 1loss/FBt4/assignment_loss‚0.17679355035238953 /loss/FBt3/detection_loss‚3.3324320554733275 /loss/FBt4/detection_loss‚0.9610858009827714 2loss/SRqqt2/assignment_loss‚1.6590996887002671 /loss/FRt1/detection_loss‚2.9003520647684735 .loss/FBt1/detection_loss‚2.227545190941204 % _timestamp‚1.7885272064444852e+09 _step‚312 0loss/FRt2/assignment_loss‚2.8230631223746707 2loss/SRqqt3/assignment_loss‚1.1523419902423175 1loss/SRqqt4/detection_loss‚2.6610685609005116 &loss/total_loss‚44.866846654060325 1loss/SRqqt4/assignment_loss‚1.986108954306002 0loss/FBt1/assignment_loss‚0.3567955703395207 /loss/FRt3/detection_loss‚1.5349855144054463 1loss/SRqqt2/detection_loss‚2.4261492558888027 /loss/FRt1/assignment_loss‚3.559944959481557 0loss/SRqqt3/detection_loss‚0.955813426192913 .loss/FBt2/detection_loss‚2.500784151694354 ,loss/FRt2/detection_loss‚2.9334745321955 /loss/FRt4/detection_loss‚2.0111504563531124 epoch‚3 trainer/global_step‚3799 0loss/FRt3/assignment_loss‚1.6332194369572859 0loss/FRt4/assignment_loss‚3.0153496279272924 0loss/FBt2/assignment_loss‚0.4104302875861688 1loss/FBt3/assignment_loss‚0.03380341596694052 1loss/SRqqt1/detection_loss‚1.9954872783273458¸œ©]¾œú‚(:‘ ë„ëÔé¡”÷ gpu.0.gpu‚15.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚33.0 gpu.0.powerWatts‚137.658' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.531599999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0NÔ,'!û‚(:ö ë„ëÔΆü* proc÷ÿ`.memory.percent‚17.807531356811523 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 187.14453125$ disk.nvme0c0n1.in‚15218.80859375 cpu‚25.397363376701474 network.recv‚ 2.061418e+06) disk./.usagePercent‚78.94031743459642% disk./.usageGB‚1176.2677764892578( disk.nvme0c0n1.out‚122596.3115234375# proc.memory.availableMB‚178.875" proc.memory.rssMB‚ 2917.5859375 proc.cpu.threads‚17 network.sent‚ 9.863593e+06 disk.dm-1.out‚ 187.14453125 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚15218.80859375" disk.nvme0n1.in‚15218.80859375& disk.nvme0n1.out‚122596.3115234375 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.88671875# disk.dm-0.out‚122596.3115234375 disk.sda.out‚ 187.14453125$ memory_percent‚98.908233642578123dÖÔü‚(:„ ú„ëÔÛúâö gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚93.277' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.6554 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÚÕW@ý‚(:ô ú„ëԨøø) disk./.usagePercent‚78.95088747367014 disk.sda.out‚ 187.5234375" proc.memory.rssMB‚ 2917.5859375* proc.memory.percent‚17.807531356811523 cpu‚25.396784637032418 network.recv‚ 2.067415e+06# disk.dm-0.out‚122950.3173828125 disk.sda3.out‚ 187.5234375" disk.nvme0n1.in‚15218.80859375 disk.sda1.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 187.5234375 disk.drbd1260.in‚0 disk.sda3.in‚0$ disk.nvme0c0n1.in‚15218.80859375( disk.nvme0c0n1.out‚122950.3173828125& disk.nvme0n1.out‚122950.3173828125 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0$ memory_percent‚98.90689849853516 network.sent‚ 9.971613e+06$ disk./.usageGB‚1176.425277709961 disk.dm-0.in‚15218.80859375 disk.sda.in‚0 disk.drbd1260.out‚ 0.88671875% proc.memory.availableMB‚ 179.09375 proc.cpu.threads‚17d(ÖY‘þ‚(:† ‰…ëÔºìÑû gpu.0.gpu‚25.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚105.192' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.0384 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹¹ oÿ‚(:ä ‰…ëÔ÷ùáý disk.sda3.out‚188.1875$ memory_percent‚98.90844821929932" proc.memory.rssMB‚ 2917.5859375% disk.nvme0n1.out‚123234.814453125 disk.sda3.in‚0 cpu‚25.396557452314607 proc.cpu.threads‚17 network.recv‚ 2.071909e+06) disk./.usagePercent‚78.94931251531949% disk./.usageGB‚1176.4018096923828" disk.nvme0c0n1.in‚ 15338.296875 disk.drbd1260.out‚0.890625 disk.sda2.in‚0 disk.sda.in‚0( proc.memory.availableMB‚ 178.83984375 network.sent‚ 9.979398e+06' disk.nvme0c0n1.out‚123234.814453125 disk.nvme0n1.in‚ 15338.296875 disk.dm-1.in‚0 disk.dm-1.out‚188.1875 disk.sda2.out‚0" disk.dm-0.out‚123234.814453125* proc.memory.percent‚17.807531356811523 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.dm-0.in‚ 15338.296875 disk.sda.out‚188.1875ÌKÅXt€‚(:é ˜…ëÔ÷ìþ÷ disk.nvme0n1.in‚ 15338.296875 disk.sda1.in‚0 disk.drbd1260.in‚0# memory_percent‚98.9084005355835$ disk./.usageGB‚1176.430004119873 disk.sda3.out‚ 188.8203125( proc.memory.availableMB‚ 178.84765625" proc.memory.rssMB‚ 2917.5859375* proc.memory.percent‚17.807531356811523 proc.cpu.threads‚17 network.sent‚ 9.98372e+06 disk.sda.in‚0 disk.sda.out‚ 188.8203125% disk.nvme0n1.out‚123580.568359375 disk.sda1.out‚0 disk.dm-1.out‚ 188.8203125 cpu‚25.39607743312063 network.recv‚ 2.076575e+06 disk.sda2.in‚0 disk.dm-0.in‚ 15338.296875" disk.dm-0.out‚123580.568359375' disk.nvme0c0n1.out‚123580.568359375 disk.dm-1.in‚0 disk.drbd1260.out‚0.890625) disk./.usagePercent‚78.95120466700506 disk.sda2.out‚0 disk.sda3.in‚0" disk.nvme0c0n1.in‚ 15338.296875öVî_‚(:… ˜…ëÔû¤¹÷ gpu.0.gpu‚20.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚95.986' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.1972 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0·W{‚‚(:„ §…ëÔ§ÕÝõ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚92.922' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.5844 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0”cµ¥vƒ‚(:ë §…ëÔÖ¿Ø÷ network.sent‚ 9.988676e+06 network.recv‚ 2.080104e+06 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.sda.in‚0 disk.sda3.in‚0) disk.nvme0c0n1.out‚123863.79443359375 cpu‚25.395745940556793) disk./.usagePercent‚78.95120466700506 disk.dm-1.in‚0 disk.dm-1.out‚189.4375 disk.dm-0.in‚ 15338.296875 disk.sda3.out‚189.4375( proc.memory.availableMB‚ 179.08984375 proc.cpu.threads‚17$ disk./.usageGB‚1176.430004119873 disk.drbd1260.out‚ 0.89453125 disk.sda2.out‚0$ disk.dm-0.out‚123863.79443359375 disk.sda.out‚189.4375" disk.nvme0c0n1.in‚ 15338.296875 disk.sda1.in‚0$ memory_percent‚98.90692234039307 disk.nvme0n1.in‚ 15338.296875' disk.nvme0n1.out‚123863.79443359375 disk.sda1.out‚0" proc.memory.rssMB‚ 2917.5859375* proc.memory.percent‚17.807531356811523*€2Ù2„- _wandbruntime‚6518  _runtime‚6518Âpê¥Ç…Á _step‚313 & lr-AdamW/pg1‚0.0007917535276028627 & lr-AdamW/pg2‚0.0007917535276028627 trainer/global_step‚3849 % _timestamp‚1.7885272800068896e+09 _runtime‚6518.227286129_åifÌ†Æ trainer/global_step‚3849 % _timestamp‚1.7885272800068896e+09 _runtime‚6518.227286129 _step‚313 & lr-AdamW/pg1‚0.0007917535276028627 & lr-AdamW/pg2‚0.0007917535276028627¹^÷«“2‡- _wandbruntime‚6518  _runtime‚6518ÊŸ¾ôÞ'ˆØO G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚73278437 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚164602787897344 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚164602787897344 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7651381237760 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7782791900160 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚149336255844048 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚149336238804688 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚71648714 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚156965941956736 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚43804318 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚156951388656640 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚698 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚166238755576320 _step‚314 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚73278439 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7651147569152 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚164602536225792 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284327936 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7651381237760 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27844396 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27843698 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚174021547476480 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚174021855744512 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 trainer/global_step‚3849 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7651147569152 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104½þúã'‰ÝO [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7651381237760 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27844396 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27843698 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚174021547476480 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚174021855744512 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 trainer/global_step‚3849 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7651147569152 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚73278437 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚164602787897344 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚164602787897344 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7651381237760 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7782791900160 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚149336255844048 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚149336238804688 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚71648714 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚156965941956736 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚43804318 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚156951388656640 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚698 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚166238755576320 _step‚314 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚73278439 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7651147569152 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚164602536225792 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284327936 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084º|ŒÔ.2Š- _wandbruntime‚6520  _runtime‚6520b{(‹äM M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚680 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 104726235 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 178049319 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚73296712 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚73296714 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚73296714 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7653345276928 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7784796389376 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚174065235935744 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚315 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 178049319 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚3849 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 104752605 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7653111595008 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚156990692696576 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7785076509696 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 280120320 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚149373769162112 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚164644037973504 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7653345276928 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 178022947 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71665560 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚156990692696576 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 SEDeviceStatsMonitor.on_tra’Ë´Âin_batch_end/allocation.small_pool.allocated‚ 104752605 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚149373786201472 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 104726235 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚73296712 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚157005430803354 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚157005184782702 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7631644601882 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚678 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚166280463486464 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚6520.526776961 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 178022947 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚166280439546368 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0ÝPÂÚï&ŒéM JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚71666240 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27851120 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚174065539996160 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚166280439546368 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚680 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 104726235 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 178049319 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚73296712 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚73296714 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚73296714 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7653345276928 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7784796389376 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚174065235935744 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚315 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 178049319 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚3849 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 104752605 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7653111595008 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚156990692696576 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7785076509696 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 280120320 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚149373769162112 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚164644037973504 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7653345276928 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 178022947 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71665560 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚156990692696576 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 104752605 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚149373786201472 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 104726235 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚73296712 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚157005430803354 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚157005184782702 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7631644601882 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚678 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚166280463486464 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚6520.526776961 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 178022947 PDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7749363275776 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚74192189 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7727624153196 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7749596944384 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚697 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 318753792 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 % _timestamp‚1.7885273664485776e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚158915690506240 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚699 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 106063839 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 106037495 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚168322954078720 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7882671507456 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚166665053782016 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 294813696 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚74192187 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7749363275776 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚72535262 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 106063839 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28190846 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _step‚318 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚158915690506240 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0’žÒ°å'¡ßO H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 294813696 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚74192187 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7749363275776 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚72535262 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 106063839 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28190846 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _step‚318 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚158915690506240 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 _runtime‚6604.668987254 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚176205944339968 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7882966321152 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚158940151848000 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7749363275776 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚74192189 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7727624153196 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7749596944384 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚697 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 318753792 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 % _timestamp‚1.7885273664485776e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚158915690506240 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚699 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 106063839 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 106037495 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚168322954078720 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7882671507456 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚166665053782016 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309¾ß±å2¢- _wandbruntime‚6606  _runtime‚6606ÌÁû è&£âM UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7751327301632 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚166706555529728 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 180301021 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚151250287033312 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7884959932416 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚151250304072672 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 180301021 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28198284 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7884658840576 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚158954994546176 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7751560983552 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚176249621921280 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 106064187 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 180274649 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚707 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚74210464 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72552797 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚168364638048768 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚74210462 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚319 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 106090557 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚705 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚44355218 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 _runtime‚6606.175322953 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚158954994546176 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚166706303844864 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚158954976543232 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 # _timestamp‚1.78852736795039e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7751560983552 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28197579 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 301091840 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 trainer/global_step‚3899 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7751327301632 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 106090557 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚166706555529728 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 180301021 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28198284 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7884658840576 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚158954994546176 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7751560983552 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚176249621921280 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 106064187 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 180274649 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚707 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚74210464 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72552797 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚168364638048768 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚74210462 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚319 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 106090557 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚705 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚44355218 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 _runtime‚6606.175322953 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚158954994546176 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚166706303844864 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚158954976543232 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 # _timestamp‚1.78852736795039e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7751560983552 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28197579 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 301091840 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 trainer/global_step‚3899 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7751327301632 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 106090557 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚166706555529728 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 180301021 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_en¤ÙÖ×zd/requested_bytes.all.peak‚ 6741059457 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚151250287033312 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7884959932416 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2¿$ÖÛÿ2¥- _wandbruntime‚6606  _runtime‚6606¨ ŽÂT¦Î 1loss/FBt1/assignment_loss‚0.22857498071022878 .loss/FBt3/detection_loss‚2.554787442088127 % _timestamp‚1.7885273679591627e+09 trainer/global_step‚3899 /loss/FRt2/assignment_loss‚4.125475520673006 .loss/FRt2/detection_loss‚3.279396176338196 /loss/FRt4/detection_loss‚2.2482841104446423 0loss/SRqqt2/detection_loss‚2.440026141979076 1loss/SRqqt3/detection_loss‚1.1178548257237515 1loss/SRqqt4/detection_loss‚2.7082926258444786 2loss/FBt3/assignment_loss‚0.007625807190407841 /loss/FRt3/assignment_loss‚1.960346017565046 0loss/FBt4/assignment_loss‚0.3185194683690697 .loss/FRt1/detection_loss‚2.894786472978263 .loss/FBt2/detection_loss‚3.013134940104051 epoch‚3 .loss/FRt1/assignment_loss‚3.02423914342091 0loss/FRt4/assignment_loss‚2.9963207784153165 2loss/SRqqt2/assignment_loss‚1.4143365668477836 _step‚320 2loss/SRqqt3/assignment_loss‚1.1895943869449264 .loss/FBt1/detection_loss‚2.605361146586282 _runtime‚6606.179863861 .loss/SRqqt4/assignment_loss‚2.166135199368 .loss/FRt3/detection_loss‚1.937727733382157 /loss/FBt4/detection_loss‚0.8576231178912249 %loss/total_loss‚46.20999811019327 2loss/SRqqt1/assignment_loss‚1.3566936895561714 1loss/FBt2/assignment_loss‚0.13919207887348312 1loss/SRqqt1/detection_loss‚1.6256697388986747Mìv§Y§Ó /loss/FRt3/assignment_loss‚1.960346017565046 0loss/FBt4/assignment_loss‚0.3185194683690697 .loss/FRt1/detection_loss‚2.894786472978263 .loss/FBt2/detection_loss‚3.013134940104051 epoch‚3 .loss/FRt1/assignment_loss‚3.02423914342091 0loss/FRt4/assignment_loss‚2.9963207784153165 2loss/SRqqt2/assignment_loss‚1.4143365668477836 _step‚320 2loss/SRqqt3/assignment_loss‚1.1895943869449264 .loss/FBt1/detection_loss‚2.605361146586282 _runtime‚6606.179863861 .loss/SRqqt4/assignment_loss‚2.166135199368 .loss/FRt3/detection_loss‚1.937727733382157 /loss/FBt4/detection_loss‚0.8576231178912249 %loss/total_loss‚46.20999811019327 2loss/SRqqt1/assignment_loss‚1.3566936895561714 1loss/FBt2/assignment_loss‚0.13919207887348312 1loss/SRqqt1/detection_loss‚1.6256697388986747 1loss/FBt1/assignment_loss‚0.22857498071022878 .loss/FBt3/detection_loss‚2.554787442088127 % _timestamp‚1.7885273679591627e+09 trainer/global_step‚3899 /loss/FRt2/assignment_loss‚4.125475520673006 .loss/FRt2/detection_loss‚3.279396176338196 /loss/FRt4/detection_loss‚2.2482841104446423 0loss/SRqqt2/detection_loss‚2.440026141979076 1loss/SRqqt3/detection_loss‚1.1178548257237515 1loss/SRqqt4/detection_loss‚2.7082926258444786 2loss/FBt3/assignment_loss‚0.007625807190407841À`nr¨‚(:„ †ëÔ×æÍø gpu.0.gpu‚6.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚117.005' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.401 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0÷x{Ø©‚(:ö †ëÔÿöŸù disk.sda1.in‚0 disk.sda.in‚0 disk.sda3.out‚ 193.3046875$ memory_percent‚98.90720844268799( proc.memory.availableMB‚ 179.04296875" proc.memory.rssMB‚ 2917.5859375* proc.memory.percent‚17.807531356811523 network.sent‚ 1.0130199e+07( disk./.usagePercent‚78.9579169285453 disk.dm-1.out‚ 193.3046875 disk.drbd1260.in‚0 disk.drbd1260.out‚0.90625 disk.sda2.in‚0 disk.dm-0.in‚15600.41796875 disk.sda.out‚ 193.3046875 network.recv‚ 2.110091e+06% disk./.usageGB‚1176.5300216674805$ disk.dm-0.out‚125978.96337890625 disk.sda3.in‚0$ disk.nvme0c0n1.in‚15600.41796875) disk.nvme0c0n1.out‚125978.96337890625" disk.nvme0n1.in‚15600.41796875' disk.nvme0n1.out‚125978.96337890625 disk.sda1.out‚0 disk.dm-1.in‚0 disk.sda2.out‚0 cpu‚25.3938129472672 proc.cpu.threads‚17O$«¯‘ª‚(:† Ÿ†ëÔ…’°÷ gpu.0.gpu‚23.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚103.361' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.6722 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬HQ-€«‚(:õ Ÿ†ëÔ‰ýïù disk.sda.out‚ 193.71875 disk.sda1.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 193.71875 disk.drbd1260.in‚0$ memory_percent‚98.90570640563965 network.sent‚ 1.0237978e+07 network.recv‚ 2.115494e+06) disk./.usagePercent‚78.90903920208555 disk.sda3.in‚0' disk.nvme0n1.out‚126497.65185546875 disk.sda1.in‚0 disk.drbd1260.out‚ 0.91015625' proc.memory.availableMB‚ 179.2890625 disk.sda2.out‚0 disk.dm-0.in‚15600.41796875$ disk.nvme0c0n1.in‚15600.41796875 cpu‚25.393485172283427 proc.cpu.threads‚17% disk./.usageGB‚1175.8017082214355$ disk.dm-0.out‚126497.65185546875 disk.sda3.out‚ 193.71875) disk.nvme0c0n1.out‚126497.65185546875" disk.nvme0n1.in‚15600.41796875" proc.memory.rssMB‚ 2917.5859375* proc.memory.percent‚17.807531356811523 disk.sda2.in‚0 disk.sda.in‚0‡é¾¬‚(:„ ®†ëÔ™Ô÷ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚92.744' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.5488 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0æ†Aˆ­‚(:ý ®†ëԬˎø disk.sda2.out‚0 disk.dm-0.in‚15600.41796875 disk.sda.out‚ 194.31640625 disk.sda3.out‚ 194.31640625% disk./.usageGB‚1175.8251876831055 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.91015625 disk.sda3.in‚0' disk.nvme0n1.out‚126860.39404296875 disk.sda1.out‚0* proc.memory.percent‚17.807531356811523$ disk.dm-0.out‚126860.26904296875$ disk.nvme0c0n1.in‚15600.41796875" disk.nvme0n1.in‚15600.41796875" proc.memory.rssMB‚ 2917.5859375 proc.cpu.threads‚17 disk.dm-1.out‚ 194.31640625 disk.sda2.in‚0 disk.sda.in‚0) disk.nvme0c0n1.out‚126860.39404296875 disk.sda1.in‚0$ memory_percent‚98.90573024749756( proc.memory.availableMB‚ 179.28515625 cpu‚25.39325221329551 network.sent‚ 1.0243189e+07 network.recv‚ 2.118953e+06( disk./.usagePercent‚78.9106149284588 disk.drbd1260.in‚0=9:윮‚(:‘ ½†ëÔ§ªŽø gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚125.733' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.146600000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0@î½ó{¯‚(:𠽆ëÔ¨÷Ÿù disk.sda1.in‚0 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda.out‚ 194.65625" proc.memory.rssMB‚ 2917.5859375* proc.memory.percent‚17.807531356811523) disk./.usagePercent‚78.90903997010815 disk.drbd1260.out‚ 0.9140625 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 194.65625$ disk.nvme0c0n1.in‚15760.05078125& disk.nvme0n1.out‚127147.3623046875( proc.memory.availableMB‚ 179.27734375 disk.dm-0.in‚15760.05078125 network.sent‚ 1.0249112e+07% disk./.usageGB‚1175.8017196655273 disk.sda1.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 194.65625" disk.nvme0n1.in‚15760.05078125$ memory_percent‚98.90577793121338# disk.dm-0.out‚127147.3623046875( disk.nvme0c0n1.out‚127147.3623046875 cpu‚25.39296032937225 proc.cpu.threads‚17 disk.sda2.in‚0 network.recv‚ 2.12371e+06gs=œ°‚(:‘ ̆ëÔ™à¢÷ gpu.0.gpu‚22.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚112.077' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.415399999999998 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0+kÜO†±‚(:û ̆ëÔ·£ø disk.sda3.in‚0& disk.nvme0n1.out‚127488.5185546875 disk.dm-1.out‚ 195.30859375$ memory_percent‚98.90573024749756" proc.memory.rssMB‚ 2917.5859375 proc.cpu.threads‚17 network.sent‚ 1.0255312e+07 disk.sda2.in‚0" disk.nvme0n1.in‚15760.05078125 disk.sda1.in‚0 cpu‚25.392760643891258 network.recv‚ 2.128396e+06 disk.dm-0.in‚15760.05078125# disk.dm-0.out‚127488.5185546875 disk.sda.in‚0 disk.sda.out‚ 195.30859375( disk.nvme0c0n1.out‚127488.5185546875( proc.memory.availableMB‚ 179.28515625* proc.memory.percent‚17.807531356811523% disk./.usageGB‚1175.8252182006836 disk.sda3.out‚ 195.30859375$ disk.nvme0c0n1.in‚15760.05078125 disk.sda1.out‚0 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.9140625) disk./.usagePercent‚78.91061697651907 disk.sda2.out‚0˜¥ÈE‘²‚(:† Û†ëÔš“Ô÷ gpu.0.gpu‚23.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚101.077' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.2154 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚08ûV]p³‚(:å Û†ëÔ¿ý¾ø( disk./.usagePercent‚78.9106156964814 disk.dm-1.in‚0 disk.sda2.in‚0 disk.dm-0.out‚ 127701.03125 disk.sda3.out‚ 195.7890625# disk.nvme0c0n1.out‚ 127701.03125 proc.cpu.threads‚17% disk./.usageGB‚1175.8251991271973 disk.sda1.out‚0 disk.dm-1.out‚ 195.7890625 disk.drbd1260.in‚0 disk.sda2.out‚0 disk.dm-0.in‚15760.05078125 disk.sda.in‚0$ disk.nvme0c0n1.in‚15760.05078125 network.sent‚ 1.0260351e+07 network.recv‚ 2.13192e+06 disk.sda.out‚ 195.7890625 disk.sda3.in‚0$ memory_percent‚98.90575408935547% proc.memory.availableMB‚ 179.28125" proc.memory.rssMB‚ 2917.5859375* proc.memory.percent‚17.807531356811523 disk.sda1.in‚0 disk.drbd1260.out‚ 0.91796875" disk.nvme0n1.in‚15760.05078125! disk.nvme0n1.out‚ 127701.03125 cpu‚25.392515428264133 ,–î2´- _wandbruntime‚6690  _runtime‚6690Xi ƵÀ & lr-AdamW/pg1‚0.0007824562903823501 & lr-AdamW/pg2‚0.0007824562903823501 trainer/global_step‚3949 $ _timestamp‚1.788527452333738e+09 _runtime‚6690.554572621 _step‚321‰"Ç˶Š_runtime‚6690.554572621 _step‚321 & lr-AdamW/pg1‚0.0007824562903823501 & lr-AdamW/pg2‚0.0007824562903823501 trainer/global_step‚3949 $ _timestamp‚1.788527452333738e+09Á[¤2·- _wandbruntime‚6690  _runtime‚6690UôMß'¸ÙO KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚160914853780152 $ _timestamp‚1.788527452337554e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 182481384 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 107401791 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚73422067 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 _step‚322 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚73422761 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚160879992355840 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 182507730 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚153089274547088 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚75105937 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44884518 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7825333212620 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7847578982400 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚75105939 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 trainer/global_step‚3949 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7847812651008 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚160914607759708 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7982458182656 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚168727571338240 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚160879992355840 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚178389921128960 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚75105939 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 Bûpä'¹ÞO JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚694 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28538243 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚692 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28537551 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚170407176521216 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 310365184 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚44884516 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚153089291586448 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚160914853780152 $ _timestamp‚1.788527452337554e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 182481384 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 107401791 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚73422067 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 _step‚322 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚73422761 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚160879992355840 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 182507730 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚153089274547088 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚75105937 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44884518 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7825333212620 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7847578982400 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚75105939 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 trainer/global_step‚3949 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7847812651008 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚160914607759708 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7982458182656 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚168727571338240 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚160879992355840 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚178389921128960 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚75105939 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚160919296395776 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.pVªc±G$eak‚ 6458321920 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 182526351 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚73440407 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚160919278392832 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚6692.061310256 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚3949 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚75124214 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 107402139 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚170448836551168 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚160919278392832 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚160954096606118 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 182552723 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73439699 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7849776690176 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚44895318 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚153126821943872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚168769073085952 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28545087 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7849776690176 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚708 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚706 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28544381 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 292703232 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104-ádï&¼éM D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚3949 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚75124214 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 107402139 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚170448836551168 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚160919278392832 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚160954096606118 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 182552723 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73439699 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7849776690176 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚44895318 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚153126821943872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚168769073085952 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28545087 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7849776690176 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚708 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚706 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28544381 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 292703232 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7827520682898 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚75124212 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 107428509 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚160919296395776 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 182526351 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚73440407 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚160919278392832 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚6692.061310256 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚154965792418288 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚326 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 184733086 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 108713399 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚180574072464896 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 108739743 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚180573772585472 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7923271253128 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚170790088894464 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8082421501952 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚162844294205440 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚154965809457648 _runtime‚6767.358709288 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚?°sâ 7946028357632 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚172491375023616 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚76019689 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28884877 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 275939328 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚172491351083520 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚76019687 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 108739743 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8082697441280 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚45424616 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7945794689024 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚162844294205440 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7946028357632 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚74310194 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚170790340566016 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚162844312208384 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚162844312208384 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 184759432 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚701 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7945794689024 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚170790088894464 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚74309493 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚162889309691860 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 108713399 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚76019687 % _timestamp‚1.7885275291391075e+09 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28885576 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚699 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 trainer/global_step‚3999 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 299879424 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608½£Ï½å'ÏßO E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚74310194 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚170790340566016 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚162844312208384 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚162844312208384 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 184759432 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚701 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7945794689024 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚170790088894464 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚74309493 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚162889309691860 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 108713399 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚76019687 % _timestamp‚1.7885275291391075e+09 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28885576 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚699 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 trainer/global_step‚3999 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 299879424 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7923500234212 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚154965792418288 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚326 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 184733086 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 108713399 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚180574072464896 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 108739743 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚180573772585472 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7923271253128 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚170790088894464 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8082421501952 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚162844294205440 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚154965809457648 _runtime‚6767.358709288 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7946028357632 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚172491375023616 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚76019689 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28884877 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 275939328 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚172491351083520 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚76019687 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 108739743 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8082697441280 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚45424616 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7945794689024 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚162844294205440 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7946028357632 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792Æžœ`2Ð- _wandbruntime‚6768  _runtime‚6768æíê&ÑäM GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚170831590642176 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 108740091 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚162928798538478 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45435418 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28892529 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 108766461 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚76037962 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7947992396800 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚74327949 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚162883598245376 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚678 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚155003322775712 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7925229742114 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚45435420 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8084704355328 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 184778053 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚170831590642176 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚180617763348992 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 % _timestamp‚1.7885275301381376e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28891851 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚172533035053568 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 288508928 _runtime‚6768.358380395 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚162883580242432 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7947758714880 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚162928552517826 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚172533058993664 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7925458723406 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 184804425 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 _step‚327 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 184804425 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 184778053 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74327269 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚162883598245376 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚162883580242432 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7947758714880 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚76037964 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 trainer/global_step‚3999 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚162883598245376 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚678 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚155003322775712 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7925229742114 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚45435420 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8084704355328 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 184778053 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚170831590642176 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚180617763348992 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 % _timestamp‚1.7885275301381376e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28891851 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚172533035053568 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 288508928 _runtime‚6768.358380395 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚162883580242432 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7947758714880 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚162928552517826 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚172533058993664 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7925458723406 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 184804425 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 _step‚327 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 184804425 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 184778053 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74327269 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚162883598245376 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 WCDeviceStatsMonitor.on_train$XT+õ_batch_end/active_bytes.large_pool.freed‚162883580242432 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7947758714880 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚76037964 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 trainer/global_step‚3999 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45435418 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28892529 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 108766461 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚76037962 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7947992396800 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚74327949Çêëù_2Ó- _wandbruntime‚6768  _runtime‚6768îÞ‘JUÔÏ 0loss/SRqqt3/detection_loss‚1.086663085907217 .loss/FBt4/detection_loss‚1.068375829231715 %loss/total_loss‚47.88834802557339 epoch‚3 0loss/FRt3/assignment_loss‚1.9504928243787665 /loss/FRt4/assignment_loss‚3.096920414517323 trainer/global_step‚3999 _runtime‚6768.361244453 1loss/SRqqt1/assignment_loss‚1.557137160934508 1loss/FBt3/assignment_loss‚0.07629942676673333 .loss/FRt3/detection_loss‚1.691924162209034 /loss/FRt4/detection_loss‚2.2565538504471383 0loss/SRqqt1/detection_loss‚2.136128318309784 /loss/FBt2/detection_loss‚2.2003091545332047 /loss/FRt2/detection_loss‚3.1262023082146277 /loss/FRt2/assignment_loss‚4.320969334015479 1loss/SRqqt2/assignment_loss‚1.825806076553735 /loss/FRt1/detection_loss‚2.9145511013490184 0loss/SRqqt4/detection_loss‚2.731450843811035 /loss/FBt1/detection_loss‚2.7840067644913993 /loss/FBt2/assignment_loss‚0.545762509498759 % _timestamp‚1.7885275301417384e+09 2loss/SRqqt3/assignment_loss‚1.0036157736734999 1loss/SRqqt2/detection_loss‚2.7901634573936462 1loss/SRqqt4/assignment_loss‚2.085471372678876 1loss/FBt1/assignment_loss‚0.43783487131069404 0loss/FBt4/assignment_loss‚0.2705491318400827 .loss/FBt3/detection_loss‚2.358976185321808 _step‚328 0loss/FRt1/assignment_loss‚3.5721840681853116™Ìe ZÕÔ 2loss/SRqqt3/assignment_loss‚1.0036157736734999 1loss/SRqqt2/detection_loss‚2.7901634573936462 1loss/SRqqt4/assignment_loss‚2.085471372678876 1loss/FBt1/assignment_loss‚0.43783487131069404 0loss/FBt4/assignment_loss‚0.2705491318400827 .loss/FBt3/detection_loss‚2.358976185321808 _step‚328 0loss/FRt1/assignment_loss‚3.5721840681853116 0loss/SRqqt3/detection_loss‚1.086663085907217 .loss/FBt4/detection_loss‚1.068375829231715 %loss/total_loss‚47.88834802557339 epoch‚3 0loss/FRt3/assignment_loss‚1.9504928243787665 /loss/FRt4/assignment_loss‚3.096920414517323 trainer/global_step‚3999 _runtime‚6768.361244453 1loss/SRqqt1/assignment_loss‚1.557137160934508 1loss/FBt3/assignment_loss‚0.07629942676673333 .loss/FRt3/detection_loss‚1.691924162209034 /loss/FRt4/detection_loss‚2.2565538504471383 0loss/SRqqt1/detection_loss‚2.136128318309784 /loss/FBt2/detection_loss‚2.2003091545332047 /loss/FRt2/detection_loss‚3.1262023082146277 /loss/FRt2/assignment_loss‚4.320969334015479 1loss/SRqqt2/assignment_loss‚1.825806076553735 /loss/FRt1/detection_loss‚2.9145511013490184 0loss/SRqqt4/detection_loss‚2.731450843811035 /loss/FBt1/detection_loss‚2.7840067644913993 /loss/FBt2/assignment_loss‚0.545762509498759 % _timestamp‚1.7885275301417384e+09È›¶O|‘Ö‚(:† µ‡ëÔ„Îðõ gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚33.0 gpu.0.powerWatts‚137.077' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.4154 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~îbw€×‚(:õ µ‡ëÔòÓ„ø) disk./.usagePercent‚78.77380373413818! disk.nvme0n1.in‚ 15938.3203125& disk.nvme0n1.out‚129262.3994140625 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 15938.3203125 disk.sda3.out‚ 198.6640625$ memory_percent‚98.91035556793213% disk./.usageGB‚1173.7866020202637( disk.nvme0c0n1.out‚129262.3994140625 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.9296875# disk.dm-0.out‚129262.3994140625 disk.sda.in‚0( proc.memory.availableMB‚ 178.52734375* proc.memory.percent‚17.807531356811523 network.sent‚ 1.0396813e+07 network.recv‚ 2.158195e+06# disk.nvme0c0n1.in‚ 15938.3203125 disk.sda2.in‚0" proc.memory.rssMB‚ 2917.5859375 cpu‚25.390348580958875 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.dm-1.out‚ 198.6640625 disk.sda2.out‚0 disk.sda.out‚ 198.6640625 disk.sda3.in‚0Ò‰»Ø‚(:„ ćëÔñÕßö gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚94.735' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.947 gpu.0.smClock‚1290 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Üp4J|Ù‚(:ñ ćëÔü¢Ê÷! disk.nvme0c0n1.in‚ 16069.84375 disk.nvme0n1.in‚ 16069.84375$ memory_percent‚98.90880584716797% proc.memory.availableMB‚ 178.78125 proc.cpu.threads‚17 network.recv‚ 2.164738e+06) disk.nvme0c0n1.out‚129553.38134765625 disk.sda1.out‚0 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.9296875 disk.dm-0.in‚ 16069.84375 network.sent‚ 1.0505615e+07 disk.sda.in‚0 disk.drbd1260.in‚0 disk.sda2.out‚0" proc.memory.rssMB‚ 2917.5859375 cpu‚25.389715965287788 disk.sda.out‚ 199.03515625 disk.sda3.in‚0' disk.nvme0n1.out‚129553.38134765625 disk.sda1.in‚0 disk.dm-1.out‚ 199.03515625 disk.sda2.in‚0$ disk.dm-0.out‚129553.38134765625* proc.memory.percent‚17.807531356811523) disk./.usagePercent‚78.77234679526082$ disk./.usageGB‚1173.764892578125 disk.sda3.out‚ 199.03515625„ûܲ‘Ú‚(:† Ó‡ëÔÆš½ö gpu.0.gpu‚37.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚110.179' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.0358 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆšÇÂzÛ‚(:ï Ó‡ëÔÛ•–ø' disk.nvme0c0n1.out‚129746.595703125$ memory_percent‚98.90882968902588 proc.cpu.threads‚17 network.recv‚ 2.168206e+06) disk./.usagePercent‚78.77392328965668% disk./.usageGB‚1173.7883834838867 disk.dm-1.in‚0 disk.dm-1.out‚ 199.51953125 disk.drbd1260.out‚ 0.93359375 disk.sda.in‚0 disk.sda3.in‚0 network.sent‚ 1.0510692e+07 disk.nvme0n1.in‚ 16069.84375 disk.sda1.out‚0 disk.sda3.out‚ 199.51953125! disk.nvme0c0n1.in‚ 16069.84375( proc.memory.availableMB‚ 178.77734375" proc.memory.rssMB‚ 2917.5859375* proc.memory.percent‚17.807531356811523 disk.drbd1260.in‚0 disk.dm-0.in‚ 16069.84375 disk.sda.out‚ 199.51953125 cpu‚25.38900520740836% disk.nvme0n1.out‚129746.595703125 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0" disk.dm-0.out‚129746.595703125í³YkÜ‚(:… â‡ëÔÇÇÑö gpu.0.gpu‚22.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚102.835' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.567 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0pä…qÝ‚(:æ â‡ëÔÕÖˆú network.recv‚ 2.171973e+06) disk./.usagePercent‚78.77391509741558' disk.nvme0c0n1.out‚129939.197265625 disk.nvme0n1.in‚ 16069.84375 disk.sda1.out‚0 disk.drbd1260.out‚ 0.93359375% disk.nvme0n1.out‚129939.197265625 disk.drbd1260.in‚0 disk.sda.in‚0 disk.sda3.out‚ 199.84375' proc.memory.availableMB‚ 178.2890625* proc.memory.percent‚17.807531356811523 cpu‚25.388832890242366 proc.cpu.threads‚17% disk./.usageGB‚1173.7882614135742 disk.dm-1.out‚ 199.84375 disk.sda2.out‚0 disk.dm-0.in‚ 16069.84375" disk.dm-0.out‚129939.197265625 disk.sda.out‚ 199.84375 disk.sda3.in‚0" proc.memory.rssMB‚ 2917.5859375 network.sent‚ 1.0515888e+07! disk.nvme0c0n1.in‚ 16069.84375 disk.sda1.in‚0 disk.dm-1.in‚0 disk.sda2.in‚0$ memory_percent‚98.91180992126465ªæ€2Þ- _wandbruntime‚6828  _runtime‚6828<,v¢ÆßÀ & lr-AdamW/pg1‚0.0007728898032067148 & lr-AdamW/pg2‚0.0007728898032067148 trainer/global_step‚4049 $ _timestamp‚1.788527590248427e+09 _runtime‚6828.469696597 _step‚329mú3ËàÅ & lr-AdamW/pg1‚0.0007728898032067148 & lr-AdamW/pg2‚0.0007728898032067148 trainer/global_step‚4049 $ _timestamp‚1.788527590248427e+09 _runtime‚6828.469696597 _step‚329ɉOÚI2á- _wandbruntime‚6828  _runtime‚6828J…9Bà'âÚO UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚164863519583124 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚76933437 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚75195599 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚164808596055040 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8044010395648 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚174575549585920 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 186984788 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚76933439 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29230883 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29231559 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 187011134 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284327936 trainer/global_step‚4049 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚172852606450688 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 % _timestamp‚1.7885275902521193e+09 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 110077695 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚678 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8044244064256 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268032 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚156842310289488 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8021209293636 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8044244064256 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _step‚330 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚676 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚182757733345792 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8182468087808 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _runtime‚6828.473652868 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚45964718 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚172852858122240 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚172852606450688 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 % _timestamp‚1.7885275902521193e+09 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 110077695 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚678 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8044244064256 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268032 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚156842310289488 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8021209293636 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8044244064256 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _step‚330 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚676 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚182757733345792 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8182468087808 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _runtime‚6828.473652868 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚45964718 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚172852858122240 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚76933437 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚75195599 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚164808596055040 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8044010395648 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚174575549585920 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 186984788 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚76933439 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29230883 ICDG‘׌6eviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29231559 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 187011134ÊJ:°ƒ2ä- _wandbruntime‚6830  _runtime‚6830vñúˆê&åäM M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚76951712 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚164847900094976 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 110078043 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚164847900094976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚45975520 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚648 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚172894108198400 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306157568 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚174617233555968 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚172894108198400 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚156879840646912 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75213306 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8023396763914 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 187056127 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8045974421504 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚164903254450186 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 110078043 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 trainer/global_step‚4049 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 187029755 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8023167782622 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚6830.835647618 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 110104413 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚650 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 187029755 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8184469255680 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚75213956 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚172893856513536 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8046208103424 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 187056127 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚172894108198400 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚156879840646912 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75213306 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8023396763914 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 187056127 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8045974421504 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚164903254450186 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 110078043 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 trainer/global_step‚4049 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 187029755 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8023167782622 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚6830.835647618 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 110104413 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚650 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 187029755 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8184469255680 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚75213956 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚172893856513536 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8046208103424 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 187056127 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚76951712 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚164847900094976 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 110078043 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104Ë>M)É2ç- _wandbruntime‚6830  _runtime‚6830é·T=]è× 0loss/FRt1/assignment_loss‚3.1044593185186384 1loss/SRqqt2/assignment_loss‚2.097729365934025 1loss/SRqqt1/detection_loss‚1.5242940263365798 &loss/total_loss‚46.161494042053235 0loss/FRt4/assignment_loss‚3.0704367728460404 1loss/FBt2/assignment_loss‚0.20671012779894227 .loss/FRt2/assignment_loss‚4.72305546076067 2loss/SRqqt1/assignment_loss‚1.4230881336514118 1loss/FBt4/assignment_loss‚0.21194972667317338 /loss/FBt2/detection_loss‚3.1659968972206114 trainer/global_step‚4049 1loss/SRqqt3/assignment_loss‚1.187049744746593 1loss/SRqqt4/assignment_loss‚2.059048442170024 /loss/FRt1/detection_loss‚2.6915254279971124 /loss/FRt4/detection_loss‚2.0256655676556483 _step‚332 1loss/FBt1/assignment_loss‚0.37370400328654796 1loss/FBt3/assignment_loss‚0.03269935492903642 /loss/FBt1/detection_loss‚2.6206021417271006 /loss/FBt3/detection_loss‚2.4838098883628845 0loss/FRt3/assignment_loss‚1.6524479619243688 /loss/FRt2/detection_loss‚2.8539018707890667 0loss/SRqqt4/detection_loss‚2.050385681065646 /loss/FBt4/detection_loss‚0.5567800229042769 epoch‚3 % _timestamp‚1.7885275926213305e+09 .loss/FRt3/detection_loss‚2.024055494681785 1loss/SRqqt2/detection_loss‚2.9021344130689446 1loss/SRqqt3/detection_loss‚1.1199641970041043 _runtime‚6830.842059033y)iÐbéÜ 1loss/SRqqt3/assignment_loss‚1.187049744746593 1loss/SRqqt4/assignment_loss‚2.059048442170024 /loss/FRt1/detection_loss‚2.6915254279971124 /loss/FRt4/detection_loss‚2.0256655676556483 _step‚332 1loss/FBt1/assignment_loss‚0.37370400328654796 1loss/FBt3/assignment_loss‚0.03269935492903642 /loss/FBt1/detection_loss‚2.6206021417271006 /loss/FBt3/detection_loss‚2.4838098883628845 0loss/FRt3/assignment_loss‚1.6524479619243688 /loss/FRt2/detection_loss‚2.8539018707890667 0loss/SRqqt4/detection_loss‚2.050385681065646 /loss/FBt4/detection_loss‚0.5567800229042769 epoch‚3 % _timestamp‚1.7885275926213305e+09 .loss/FRt3/detection_loss‚2.024055494681785 1loss/SRqqt2/detection_loss‚2.9021344130689446 1loss/SRqqt3/detection_loss‚1.1199641970041043 _runtime‚6830.842059033 0loss/FRt1/assignment_loss‚3.1044593185186384 1loss/SRqqt2/assignment_loss‚2.097729365934025 1loss/SRqqt1/detection_loss‚1.5242940263365798 &loss/total_loss‚46.161494042053235 0loss/FRt4/assignment_loss‚3.0704367728460404 1loss/FBt2/assignment_loss‚0.20671012779894227 .loss/FRt2/assignment_loss‚4.72305546076067 2loss/SRqqt1/assignment_loss‚1.4230881336514118 1loss/FBt4/assignment_loss‚0.21194972667317338 /loss/FBt2/detection_loss‚3.1659968972206114 trainer/global_step‚4049ÌJ¸¦3›ê‚(: ñ‡ëÔƒÇè÷ gpu.0.gpu‚19.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚33.0 gpu.0.powerWatts‚141.74' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.348000000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ìà…së‚(:è ñ‡ëÔ¥™…ú disk.sda.out‚ 200.3671875' disk.nvme0c0n1.out‚130037.013671875 disk.nvme0n1.in‚ 16069.84375% disk.nvme0n1.out‚130037.013671875 disk.sda1.in‚0$ memory_percent‚98.91207218170166" disk.dm-0.out‚130037.013671875 disk.sda1.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 200.3671875* proc.memory.percent‚17.807531356811523 cpu‚25.388530946254413 disk.sda.in‚0 disk.sda3.out‚ 200.3671875 disk.drbd1260.in‚0 disk.drbd1260.out‚0.9375( proc.memory.availableMB‚ 178.24609375" proc.memory.rssMB‚ 2917.5859375 proc.cpu.threads‚17 network.sent‚ 1.0521938e+07 network.recv‚ 2.176415e+06( disk./.usagePercent‚78.7671959236719% disk./.usageGB‚1173.6881408691406 disk.sda2.in‚0 disk.dm-0.in‚ 16069.84375 disk.sda3.in‚0! disk.nvme0c0n1.in‚ 16069.84375 disk.sda2.out‚0ÁáÓýlì‚(:á €ˆëÔÚ±Ñø disk.dm-1.in‚0 cpu‚25.387878921670588 disk.sda3.in‚0 network.recv‚ 2.183489e+06) disk./.usagePercent‚78.77248324727657 disk.sda.in‚0% disk.nvme0n1.out‚130117.982421875 disk.dm-1.out‚ 200.7734375# memory_percent‚98.9105224609375! proc.memory.availableMB‚178.5 disk.drbd1260.in‚0 disk.sda.out‚ 200.7734375 disk.nvme0n1.in‚ 16209.78125" proc.memory.rssMB‚ 2917.5859375* proc.memory.percent‚17.807531356811523 proc.cpu.threads‚17 network.sent‚ 1.0631011e+07% disk./.usageGB‚1173.7669258117676 disk.sda2.out‚0 disk.dm-0.in‚ 16209.78125" disk.dm-0.out‚130117.982421875' disk.nvme0c0n1.out‚130117.982421875 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.out‚0.9375 disk.sda2.in‚0 disk.sda3.out‚ 200.7734375! disk.nvme0c0n1.in‚ 16209.78125®„§þí‚(:… €ˆëÔßËÒù gpu.0.gpu‚8.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚114.479' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.8958 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0éò§—yî‚(:î ˆëÔ±²Ë÷ disk.sda2.out‚0 disk.sda3.in‚0! disk.nvme0c0n1.in‚ 16209.78125 disk.dm-1.out‚ 201.296875" proc.memory.rssMB‚ 2917.5859375 cpu‚25.387407259170516 disk.sda3.out‚ 201.296875 disk.drbd1260.out‚ 0.9453125 disk.sda.in‚0) disk.nvme0c0n1.out‚130347.67724609375 disk.sda1.in‚0 disk.dm-1.in‚0* proc.memory.percent‚17.807531356811523 disk.drbd1260.in‚0 disk.dm-0.in‚ 16209.78125 disk.nvme0n1.in‚ 16209.78125' disk.nvme0n1.out‚130347.67724609375' proc.memory.availableMB‚ 178.7421875 proc.cpu.threads‚17 network.sent‚ 1.0636083e+07 network.recv‚ 2.186791e+06% disk./.usageGB‚1173.4095573425293 disk.sda2.in‚0$ disk.dm-0.out‚130347.67724609375 disk.sda.out‚ 201.296875 disk.sda1.out‚0$ memory_percent‚98.90904426574707) disk./.usagePercent‚78.74849994945387ªZ€‘ï‚(:† ˆëÔð¹Ÿú gpu.0.gpu‚25.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚101.848' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.3696 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0wŠœð‚(:‘ žˆëÔìçòö gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚133.243' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.648600000000002 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~þÒÉxñ‚(:í žˆëÔ¡î«÷ disk.drbd1260.out‚ 0.9453125 disk.drbd1260.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 16209.78125$ disk.dm-0.out‚130540.32568359375# memory_percent‚98.9105224609375! proc.memory.availableMB‚178.5 cpu‚25.387075922240633 disk.sda1.out‚0 disk.dm-1.out‚ 201.75390625 disk.sda3.in‚0 disk.sda3.out‚ 201.75390625" proc.memory.rssMB‚ 2917.5859375* proc.memory.percent‚17.807531356811523 proc.cpu.threads‚17 network.recv‚ 2.190505e+06% disk./.usageGB‚1173.4096298217773! disk.nvme0c0n1.in‚ 16209.78125) disk.nvme0c0n1.out‚130540.32568359375 disk.nvme0n1.in‚ 16209.78125 disk.sda1.in‚0 disk.dm-1.in‚0 disk.sda2.in‚0 network.sent‚ 1.0641123e+07) disk./.usagePercent‚78.74850481359702' disk.nvme0n1.out‚130540.32568359375 disk.sda.in‚0 disk.sda.out‚ 201.75390625Çdü2ò- _wandbruntime‚6894  _runtime‚6894›3;ÆóÀ & lr-AdamW/pg1‚0.0007680080742214406 & lr-AdamW/pg2‚0.0007680080742214406 trainer/global_step‚4099 $ _timestamp‚1.788527656058494e+09 _runtime‚6894.279133267 _step‚333ido¡ËôÅ _runtime‚6894.279133267 _step‚333 & lr-AdamW/pg1‚0.0007680080742214406 & lr-AdamW/pg2‚0.0007680080742214406 trainer/global_step‚4099 $ _timestamp‚1.788527656058494e+09Íåæ2õ- _wandbruntime‚6894  _runtime‚6894;¦ë%öÚO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚77847189 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚174915124006912 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8142459770880 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 189262836 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 111389303 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29577939 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8282054281728 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8142459770880 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚76082071 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚76082757 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚176659748088320 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚174915375678464 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚166772915907584 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_dF˜»start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 111415647 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚174915375678464 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚174915124006912 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚158718845200048 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚158718828160688 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8282355386880 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 111415647 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚166837975494832 trainer/global_step‚4099 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚166772897904640 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 _step‚334 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 111389303 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29577255 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8142226102272 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚166772897904640 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚77847187 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚46504818 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8142226102272 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 189236490 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 325045248 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 _step‚334 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 111389303 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29577255 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8142226102272 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚166772897904640 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚77847187 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚46504818 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8142226102272 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 189236490 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 325045248 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚174915124006912 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8142459770880 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 189262836 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 111389303 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29577939 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8282054281728 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8142459770880 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚76082071 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚76082757 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚176659748088320 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚174915375678464 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚166772915907584 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 111415647 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚174915375678464 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚174915124006912 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚158718845200048 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚158718828160688 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8282355386880 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 111415647 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚166837975494832ÎÇ傼2ø- _wandbruntime‚6896  _runtime‚6896V0Ftê&ùäM OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 111415995 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8121334804422 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚174956625754624 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚166812201944576 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚176701432058368 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚77865464 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8144190128128 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8144423810048 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8284053822976 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚158756358518112 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚174956625754624 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚76100698 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8284338137600 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29585078 _step‚335 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 189307829 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 189307829 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚184985794136064 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 % _timestamp‚1.7885276581447434e+09 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚77865462 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚46515618 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 111442365 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8144423810048 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚77865462 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚46515620 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 111442365 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚174956374069760 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 trainer/global_step‚4099 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8121105823130 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 189281457 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚176701455998464 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76100035 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 111415995 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚663 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29584417 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 308254720 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚158756375557472 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚77865464 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚166812183941632 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚166812183941632 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104ué’­!úéM M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 308254720 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚158756375557472 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚77865464 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚166812183941632 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚166812183941632 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 111415995 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8121334804422 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚174956625754624 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚166812201944576 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚176701432058368 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚77865464 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8144190128128 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8144423810048 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8284053822976 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚158756358518112 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚174956625754624 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚76100698 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8284338137600 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29585078 _step‚335 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 189307829 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 189307829 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚184985794136064 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 % _timestamp‚1.7885276581447434e+09 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚77865462 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚46515618 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 C9DeviceStatsMonitor.on_train_batch_end/allocationYDc³Î.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 111442365 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8144423810048 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚77865462 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚46515620 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 111442365 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚174956374069760 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 trainer/global_step‚4099 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8121105823130 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 189281457 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚176701455998464 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76100035 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 111415995 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚663 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29584417 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚187125801422336 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 112727255 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29928159 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚168812677426984 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8382139159552 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚168812431406540 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 _runtime‚6977.434980735 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚176977641563136 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚168737217757184 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚78760937 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚78760939 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚47044918 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚187126109690368 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 112753599 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚676 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8240675477504 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚176977641563136 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚168737217757184 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚78760937 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚78760939 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚47044918 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚187126109690368 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 112753599 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚676 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8240675477504 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚187125801422336 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 112727255 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29928159 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚168812677426984 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8382139159552 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚168812431406540 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0Òðã0r2- _wandbruntime‚6979  _runtime‚6979¶J‘ê&‘äM TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚160632876389312 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 191533159 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8242405834752 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚47055720 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚168776503794176 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚47055718 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚76991757 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286411776 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚177019143310848 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚160632893428672 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚78779214 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚78779212 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚78779214 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 _step‚339 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8219272844930 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 trainer/global_step‚4149 _runtime‚6979.688408095 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8242639516672 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚177018891625984 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 112753947 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 112753947 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29936037 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚187169483997696 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚168852166273602 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚177019143310848 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8242639516672 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚178785630560768 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚168851920252950 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚168776485791232 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76991075 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 191559531 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 7/>ï&’éM T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚177019143310848 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚160632893428672 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚78779214 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚78779212 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚78779214 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 _step‚339 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8219272844930 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 trainer/global_step‚4149 _runtime‚6979.688408095 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8242639516672 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚177018891625984 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 112753947 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 112753947 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29936037 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚187169483997696 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚168852166273602 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚177019143310848 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8242639516672 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚178785630560768 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚168851920252950 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚168776485791232 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76991075 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 191559531 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚160632876389312 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 191533159 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8242405834752 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚47055720 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚168776503794176 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚47055718 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚76991757 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286411776 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0Ó„”©82“- _wandbruntime‚6979  _runtime‚6979;”ºß_”Ù &loss/total_loss‚47.908498857342245 epoch‚3 0loss/FRt2/assignment_loss‚3.9735337123274803 2loss/SRqqt2/assignment_loss‚1.7695027448629077 2loss/SRqqt3/assignment_loss‚1.1667015227794764 /loss/FRt1/detection_loss‚2.9793779952772734 /loss/FRt4/detection_loss‚1.7705268892738968 /loss/FBt4/detection_loss‚0.9618000063445356 1loss/SRqqt2/detection_loss‚2.9743386726630363 .loss/FBt3/detection_loss‚2.268749882777532 _runtime‚6979.694043107 _step‚340 0loss/FRt1/assignment_loss‚3.2734968415622054 1loss/FBt2/assignment_loss‚0.23982827580143098 0loss/FBt3/assignment_loss‚0.4557254963244001 /loss/FRt2/detection_loss‚3.7087423652410507 /loss/FRt3/detection_loss‚1.5009938359812454 1loss/SRqqt4/detection_loss‚2.8988703319004605 .loss/FBt1/detection_loss‚2.963894850677914 trainer/global_step‚4149 0loss/FRt4/assignment_loss‚3.5105689370632174 /loss/FBt4/assignment_loss‚0.274932426905666 1loss/SRqqt1/detection_loss‚2.1594421967864035 /loss/FBt2/detection_loss‚2.3806901233536855 0loss/FRt3/assignment_loss‚1.4623780625837821 2loss/SRqqt1/assignment_loss‚1.5571675021201372 2loss/SRqqt4/assignment_loss‚2.4312413867030824 1loss/SRqqt3/detection_loss‚0.9563491416629404 % _timestamp‚1.7885277414734077e+09 1loss/FBt1/assignment_loss‚0.26964565636848825Çd•Þ /loss/FRt1/detection_loss‚2.9793779952772734 /loss/FRt4/detection_loss‚1.7705268892738968 /loss/FBt4/detection_loss‚0.9618000063445356 1loss/SRqqt2/detection_loss‚2.9743386726630363 .loss/FBt3/detection_loss‚2.268749882777532 _runtime‚6979.694043107 _step‚340 0loss/FRt1/assignment_loss‚3.2734968415622054 1loss/FBt2/assignment_loss‚0.23982827580143098 0loss/FBt3/assignment_loss‚0.4557254963244001 /loss/FRt2/detection_loss‚3.7087423652410507 /loss/FRt3/detection_loss‚1.5009938359812454 1loss/SRqqt4/detection_loss‚2.8988703319004605 .loss/FBt1/detection_loss‚2.963894850677914 trainer/global_step‚4149 0loss/FRt4/assignment_loss‚3.5105689370632174 /loss/FBt4/assignment_loss‚0.274932426905666 1loss/SRqqt1/detection_loss‚2.1594421967864035 /loss/FBt2/detection_loss‚2.3806901233536855 0loss/FRt3/assignment_loss‚1.4623780625837821 2loss/SRqqt1/assignment_loss‚1.5571675021201372 2loss/SRqqt4/assignment_loss‚2.4312413867030824 1loss/SRqqt3/detection_loss‚0.9563491416629404 % _timestamp‚1.7885277414734077e+09 1loss/FBt1/assignment_loss‚0.26964565636848825 &loss/total_loss‚47.908498857342245 epoch‚3 0loss/FRt2/assignment_loss‚3.9735337123274803 2loss/SRqqt2/assignment_loss‚1.7695027448629077 2loss/SRqqt3/assignment_loss‚1.1667015227794764ÔZ÷«ö–‚(:… ‡‰ëÔ¾é¦÷ gpu.0.gpu‚26.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚102.135' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.427 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0P‰û†—‚(:û ‡‰ëÔ¥Ú÷" disk.nvme0n1.in‚16508.30078125' disk.nvme0n1.out‚132466.61083984375$ memory_percent‚98.91111850738525 cpu‚25.385100142871863 network.sent‚ 1.0782479e+07 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.9609375$ disk.dm-0.out‚132466.61083984375 disk.sda.out‚ 205.2890625) disk.nvme0c0n1.out‚132466.61083984375( proc.memory.availableMB‚ 178.40234375) disk./.usagePercent‚78.82753766751495% disk./.usageGB‚1174.5872764587402 disk.sda2.out‚0 disk.dm-0.in‚16508.30078125 disk.sda.in‚0" proc.memory.rssMB‚ 2917.5859375* proc.memory.percent‚17.807531356811523 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 205.2890625$ disk.nvme0c0n1.in‚16508.30078125 disk.sda1.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17 network.recv‚ 2.220132e+06 disk.dm-1.out‚ 205.2890625˜‚(:‘ –‰ëÔ¼öìö gpu.0.gpu‚25.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚103.076' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚20.615199999999998 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0MðzO†™‚(:û –‰ëÔŒ½ˆø disk.drbd1260.out‚ 0.9609375( proc.memory.availableMB‚ 178.15234375* proc.memory.percent‚17.807531356811523 network.sent‚ 1.0890378e+07 disk.sda2.in‚0 disk.dm-0.in‚16508.30078125$ disk.dm-0.out‚132902.86865234375 disk.sda.out‚ 205.53515625 disk.sda3.out‚ 205.53515625 disk.sda1.in‚0 disk.sda1.out‚0' disk./.usagePercent‚78.827534339417" disk.nvme0n1.in‚16508.30078125 disk.dm-1.out‚ 205.53515625 cpu‚25.384845630853324% disk./.usageGB‚1174.5872268676758 disk.sda2.out‚0$ disk.nvme0c0n1.in‚16508.30078125' disk.nvme0n1.out‚132902.86865234375 disk.dm-1.in‚0# memory_percent‚98.9126443862915" proc.memory.rssMB‚ 2917.5859375 proc.cpu.threads‚17 network.recv‚ 2.225818e+06 disk.sda.in‚0 disk.sda3.in‚0) disk.nvme0c0n1.out‚132902.86865234375 disk.drbd1260.in‚0a~$Iœš‚(:‘ ¥‰ëÔÒ¾œø gpu.0.gpu‚30.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚132.645' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.529000000000003 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘>û|›‚(:ñ ¥‰ëÔýºîø" proc.memory.rssMB‚ 2917.5859375 disk.sda.in‚0 disk.sda3.out‚206.0625 disk.sda1.in‚0 proc.cpu.threads‚17 disk.dm-0.in‚16508.30078125 network.sent‚ 1.0896384e+07 network.recv‚ 2.230126e+06% disk./.usageGB‚1175.0872268676758 disk.dm-1.in‚0 disk.drbd1260.out‚ 0.96484375 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.out‚206.0625 disk.drbd1260.in‚0 disk.sda.out‚206.0625" disk.nvme0n1.in‚16508.30078125' disk.nvme0n1.out‚132961.19482421875 disk.sda1.out‚0* proc.memory.percent‚17.807531356811523 cpu‚25.384518282381418$ disk.dm-0.out‚132961.19482421875) disk./.usagePercent‚78.86108975894494 disk.sda3.in‚0$ disk.nvme0c0n1.in‚16508.30078125) disk.nvme0c0n1.out‚132961.19482421875$ memory_percent‚98.91119003295898& proc.memory.availableMB‚ 178.390625ávu¸œ‚(:… ´‰ëÔÂæ¨÷ gpu.0.gpu‚26.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚97.863' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.5726 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ó]ùf‚(:Û ´‰ëÔÛÌÆø disk.sda.in‚0) disk.nvme0c0n1.out‚133031.86669921875 disk.nvme0n1.in‚16620.25' disk.nvme0n1.out‚133031.86669921875$ memory_percent‚98.90964031219482" proc.memory.rssMB‚ 2917.5859375 network.sent‚ 1.0902596e+07 network.recv‚ 2.234791e+06$ disk./.usageGB‚1175.099308013916 disk.sda3.in‚0 disk.sda3.out‚206.625 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.96484375 disk.sda.out‚206.625 disk.dm-1.out‚206.625( proc.memory.availableMB‚ 178.64453125 cpu‚25.3843417316845) disk./.usagePercent‚78.86190053480588 disk.sda2.out‚0$ disk.dm-0.out‚133031.86669921875 disk.nvme0c0n1.in‚16620.25* proc.memory.percent‚17.807531356811523 proc.cpu.threads‚17 disk.sda2.in‚0 disk.dm-0.in‚16620.25Òa\-‘ž‚(:† ÉëÔŠƒ†ú gpu.0.gpu‚17.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚114.994' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.9988 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡G©~Ÿ‚(:ó ÉëÔ½ÕŒü disk.sda3.out‚ 207.140625( disk.nvme0c0n1.out‚133187.5146484375" disk.nvme0n1.in‚16642.74609375 disk.sda1.in‚0 disk.dm-1.out‚ 207.140625 disk.dm-0.in‚16642.74609375 network.recv‚ 2.237985e+06) disk./.usagePercent‚78.87025636471596$ disk.nvme0c0n1.in‚16642.74609375 disk.dm-1.in‚0 disk.sda2.out‚0 disk.sda.out‚ 207.140625* proc.memory.percent‚17.807531356811523 disk.sda1.out‚0 disk.drbd1260.out‚0.96875 disk.sda2.in‚0 disk.sda.in‚0$ memory_percent‚98.91088008880615( proc.memory.availableMB‚ 178.44140625" proc.memory.rssMB‚ 2917.5859375 cpu‚25.384031866582617% disk./.usageGB‚1175.2238159179688& disk.nvme0n1.out‚133187.5146484375 disk.drbd1260.in‚0# disk.dm-0.out‚133187.5146484375 proc.cpu.threads‚17 network.sent‚ 1.0909523e+07 disk.sda3.in‚0G’U…2 - _wandbruntime‚7060  _runtime‚7060èÿóµÇ¡Á _step‚341 & lr-AdamW/pg1‚0.0007580527319689842 & lr-AdamW/pg2‚0.0007580527319689842 trainer/global_step‚4199 % _timestamp‚1.7885278220558033e+09 _runtime‚7060.275513967"¢Ì¢Æ & lr-AdamW/pg2‚0.0007580527319689842 trainer/global_step‚4199 % _timestamp‚1.7885278220558033e+09 _runtime‚7060.275513967 _step‚341 & lr-AdamW/pg1‚0.0007580527319689842Õ3åÌÏ2£- _wandbruntime‚7060  _runtime‚7060W:U4®¤ÚO UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ÍkjT2' 109171712 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚672 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 _step‚342 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 114091551 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚170787133338692 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8315023415160 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚180828145093120 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚674 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚170701519606784 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚79674687 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚4199 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8338657515520 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚179040159119360 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚79674689 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30276705 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚189309829271040 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284327936 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30277377 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚189310137539072 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8481968505856 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7885278220567136e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 114065207 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 114091551 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚79674689 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚179040410790912 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚47585016 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 193739894 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚77862395 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8481684177920 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚170786887318248 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚179040159119360 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚77861721 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552£©å'¥ßO =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚170701519606784 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚79674687 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚4199 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8338657515520 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚179040159119360 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚79674689 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30276705 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚189309829271040 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284327936 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30277377 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚189310137539072 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8481968505856 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7885278220567136e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 114065207 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 114091551 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚79674689 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚179040410790912 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚47585016 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 193739894 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚77862395 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8481684177920 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚170786887318248 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚179040159119360 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚77861721 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚672 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 _step‚342 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 114091551 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚170787133338692 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8315023415160 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚180828145093120 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚674 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624Öw2¦- _wandbruntime‚7061  _runtime‚7061]&ýê&§äM T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚179081660867072 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30284090 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8340621541376 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30283399 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8483673610240 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚79692964 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚79692962 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚180869853003264 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚170740805643776 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 114091899 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 320837632 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 114091899 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 114118269 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77879217 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 114118269 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 trainer/global_step‚4199 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚79692962 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚170826376164658 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8316981904146 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚180869829063168 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 _runtime‚7061.299261385 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 193784861 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚691 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚189353823511040 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚179081409182208 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚77879910 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚170740787640832 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚693 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚47595818 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚179081660867072 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 % _timestamp‚1.7885278230789533e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 193811233 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8317210885438 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 193811233 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚47595820 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚79692964 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚79692962 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1808698530032çiZE64 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚170740805643776 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 114091899 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 320837632 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 114091899 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 114118269 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77879217 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 114118269 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 trainer/global_step‚4199 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚79692962 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚170826376164658 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8316981904146 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚180869829063168 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 _runtime‚7061.299261385 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 193784861 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚691 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚189353823511040 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚179081409182208 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚77879910 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚170740787640832 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚693 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚47595818 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚179081660867072 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 % _timestamp‚1.7885278230789533e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 193811233 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8317210885438 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 193811233 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚47595820 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0ךۊ¹2©- _wandbruntime‚7061  _runtime‚7061çGYFVªÐ 2loss/SRqqt1/assignment_loss‚1.5324610391178646 2loss/FBt3/assignment_loss‚0.005737874629176076 0loss/FBt4/assignment_loss‚0.1637225919962733 0loss/FRt3/assignment_loss‚1.6726500036111518 #loss/total_loss‚46.529880983621 epoch‚3 .loss/FRt2/assignment_loss‚3.22766187316493 1loss/SRqqt3/assignment_loss‚1.172063131190451 /loss/SRqqt4/detection_loss‚2.75675902868572 2loss/SRqqt2/assignment_loss‚1.9687797479221927 _step‚344 /loss/FBt2/assignment_loss‚0.257548109125898 /loss/FRt1/detection_loss‚3.3069008106277105 /loss/FRt2/detection_loss‚3.3840138284783614 /loss/FRt4/detection_loss‚1.6915802888513085 % _timestamp‚1.7885278230816133e+09 _runtime‚7061.301055805 0loss/FRt4/assignment_loss‚3.4119717592284795 1loss/SRqqt1/detection_loss‚1.8091714615757402 trainer/global_step‚4199 1loss/SRqqt4/assignment_loss‚2.273069671894375 1loss/SRqqt3/detection_loss‚1.0529314390672693 /loss/FBt1/detection_loss‚2.6324025851029615 .loss/FBt3/detection_loss‚2.705151379108429 /loss/FBt4/detection_loss‚0.9996393389864674 /loss/FRt1/assignment_loss‚3.397164305051168 0loss/FBt1/assignment_loss‚0.2910428146351478 /loss/FRt3/detection_loss‚1.7512911964041085 1loss/SRqqt2/detection_loss‚3.0030022193403805 /loss/FBt2/detection_loss‚2.0631644858254328öõy[«Õ /loss/FBt2/assignment_loss‚0.257548109125898 /loss/FRt1/detection_loss‚3.3069008106277105 /loss/FRt2/detection_loss‚3.3840138284783614 /loss/FRt4/detection_loss‚1.6915802888513085 % _timestamp‚1.7885278230816133e+09 _runtime‚7061.301055805 0loss/FRt4/assignment_loss‚3.4119717592284795 1loss/SRqqt1/detection_loss‚1.8091714615757402 trainer/global_step‚4199 1loss/SRqqt4/assignment_loss‚2.273069671894375 1loss/SRqqt3/detection_loss‚1.0529314390672693 /loss/FBt1/detection_loss‚2.6324025851029615 .loss/FBt3/detection_loss‚2.705151379108429 /loss/FBt4/detection_loss‚0.9996393389864674 /loss/FRt1/assignment_loss‚3.397164305051168 0loss/FBt1/assignment_loss‚0.2910428146351478 /loss/FRt3/detection_loss‚1.7512911964041085 1loss/SRqqt2/detection_loss‚3.0030022193403805 /loss/FBt2/detection_loss‚2.0631644858254328 2loss/SRqqt1/assignment_loss‚1.5324610391178646 2loss/FBt3/assignment_loss‚0.005737874629176076 0loss/FBt4/assignment_loss‚0.1637225919962733 0loss/FRt3/assignment_loss‚1.6726500036111518 #loss/total_loss‚46.529880983621 epoch‚3 .loss/FRt2/assignment_loss‚3.22766187316493 1loss/SRqqt3/assignment_loss‚1.172063131190451 /loss/SRqqt4/detection_loss‚2.75675902868572 2loss/SRqqt2/assignment_loss‚1.9687797479221927 _step‚344Ø2ch¬‚(:Ý Ò‰ëÔ®å…ø network.sent‚ 1.0912709e+07$ disk.nvme0c0n1.in‚16642.74609375" disk.nvme0n1.in‚16642.74609375 disk.sda1.in‚0' proc.memory.availableMB‚ 178.3828125 proc.cpu.threads‚17$ disk./.usageGB‚1175.224266052246 disk.dm-1.in‚0 disk.dm-1.out‚207.5 disk.drbd1260.in‚0# disk.dm-0.out‚133600.8544921875 cpu‚25.3837790852705( disk.nvme0c0n1.out‚133600.8544921875& disk.nvme0n1.out‚133600.8544921875 disk.drbd1260.out‚0.96875 disk.sda2.out‚0 disk.dm-0.in‚16642.74609375 disk.sda.in‚0 disk.sda.out‚207.5 disk.sda3.in‚0 network.recv‚ 2.241471e+06' disk./.usagePercent‚78.870286573605 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda3.out‚207.5# memory_percent‚98.9112377166748" proc.memory.rssMB‚ 2917.5859375* proc.memory.percent‚17.807531356811523Íó%­‚(:„ Ò‰ëÔŸúÒø gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚95.263' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.0526 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0’O¤ë›®‚(: á‰ëÔ•…ÿõ gpu.0.gpu‚24.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚94.912' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚18.982400000000002 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0à“\Vz¯‚(:ï á‰ëÔˆœªü network.recv‚ 2.247941e+06% disk./.usageGB‚1175.3492469787598$ disk.dm-0.out‚133977.13427734375 disk.sda.in‚0 disk.sda3.in‚0" disk.nvme0n1.in‚16642.74609375' disk.nvme0n1.out‚133977.13427734375 disk.sda1.out‚0 network.sent‚ 1.1021427e+07 disk.dm-0.in‚16642.74609375 disk.sda.out‚207.9375 disk.sda3.out‚207.9375) disk.nvme0c0n1.out‚133977.13427734375 disk.dm-1.in‚0 disk.dm-1.out‚207.9375 disk.drbd1260.in‚0$ disk.nvme0c0n1.in‚16642.74609375 disk.sda1.in‚0 disk.drbd1260.out‚ 0.97265625 disk.sda2.in‚0 disk.sda2.out‚0& proc.memory.availableMB‚ 177.859375 proc.cpu.threads‚17( disk./.usagePercent‚78.8786741484493$ memory_percent‚98.91443252563477" proc.memory.rssMB‚ 2917.5859375* proc.memory.percent‚17.807531356811523 cpu‚25.38357120317927ÖË¡Cœ°‚(:‘ ð‰ëÔ÷ÔÅú gpu.0.gpu‚39.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.956' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.991200000000003 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µ…¨íˆ±‚(:ý ð‰ëÔôö¥‡ disk.dm-1.in‚0 disk.dm-1.out‚ 208.60546875 disk.drbd1260.in‚0 disk.drbd1260.out‚ 0.97265625& proc.memory.availableMB‚ 178.109375" proc.memory.rssMB‚ 2917.5859375% disk./.usageGB‚1175.3257789611816) disk.nvme0c0n1.out‚134203.58740234375 disk.sda1.in‚0 disk.sda2.in‚0* proc.memory.percent‚17.807531356811523 cpu‚25.383281848277385 proc.cpu.threads‚17 network.recv‚ 2.252677e+06 disk.sda3.in‚0' disk.nvme0n1.out‚134203.58740234375 disk.sda2.out‚0$ memory_percent‚98.91290664672852" disk.nvme0n1.in‚16642.74609375) disk./.usagePercent‚78.87709919009866$ disk.dm-0.out‚134203.58740234375 disk.sda.in‚0 disk.sda.out‚ 208.60546875 disk.sda3.out‚ 208.60546875$ disk.nvme0c0n1.in‚16642.74609375 disk.sda1.out‚0 network.sent‚ 1.1027696e+07 disk.dm-0.in‚16642.74609375 uŠ'²‚(:… ÿ‰ëÔÐθø gpu.0.gpu‚28.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚111.265' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.253 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0gŽÊˆ³‚(:ô ÿ‰ëÔÏñØù network.sent‚ 1.1032628e+07 disk.sda2.in‚0 disk.sda.in‚0 disk.sda.out‚ 209.109375 disk.sda3.out‚ 209.109375" disk.nvme0n1.in‚16807.94140625 proc.cpu.threads‚17( disk./.usagePercent‚78.9122254718567 disk.sda2.out‚0 disk.sda3.in‚0$ disk.nvme0c0n1.in‚16807.94140625) disk.nvme0c0n1.out‚134693.11865234375 disk.dm-1.in‚0 disk.dm-1.out‚ 209.109375$ memory_percent‚98.91138076782227 disk.drbd1260.in‚0 disk.dm-0.in‚16807.94140625$ disk.dm-0.out‚134693.11865234375' disk.nvme0n1.out‚134693.11865234375 disk.sda1.in‚0" proc.memory.rssMB‚ 2917.5859375 cpu‚25.38306592147943 network.recv‚ 2.256105e+06% disk./.usageGB‚1175.8491859436035 disk.drbd1260.out‚ 0.9765625 disk.sda1.out‚0& proc.memory.availableMB‚ 178.359375* proc.memory.percent‚17.807531356811523tèÊ›´‚(: ŽŠëÔ懸ö gpu.0.gpu‚24.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚105.24' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.048000000000002 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0z†µ‚(:û ŽŠëÔ¤’Ò÷) disk.nvme0c0n1.out‚134751.36083984375 network.sent‚ 1.1037889e+07 network.recv‚ 2.259938e+06( disk./.usagePercent‚78.9122254718567 disk.dm-0.in‚16807.94140625$ disk.dm-0.out‚134751.36083984375 disk.sda3.out‚ 209.51171875 disk.sda1.in‚0% disk./.usageGB‚1175.8491859436035 disk.sda2.out‚0 disk.sda3.in‚0$ disk.nvme0c0n1.in‚16807.94140625" disk.nvme0n1.in‚16807.94140625 disk.sda1.out‚0 disk.dm-1.out‚ 209.51171875 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.sda.in‚0' disk.nvme0n1.out‚134751.36083984375 disk.drbd1260.out‚ 0.9765625$ memory_percent‚98.91138076782227 proc.cpu.threads‚17 disk.sda.out‚ 209.51171875 disk.dm-1.in‚0& proc.memory.availableMB‚ 178.359375" proc.memory.rssMB‚ 2917.5859375* proc.memory.percent‚17.807531356811523 cpu‚25.3828725398070357¾Ý›¶‚(:… ŠëÔЃÚø gpu.0.gpu‚20.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.031' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.6062 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ú±uqz·‚(:ï ŠëÔÆ …û network.recv‚ 2.26414e+06 disk.dm-0.in‚16807.94140625 disk.sda1.out‚0 disk.drbd1260.out‚ 0.98046875& proc.memory.availableMB‚ 177.859375* proc.memory.percent‚17.807531356811523 proc.cpu.threads‚17% disk./.usageGB‚1176.1002044677734 disk.sda2.out‚0 disk.sda.in‚0$ disk.nvme0c0n1.in‚16807.94140625' disk.nvme0c0n1.out‚135150.833984375% disk.nvme0n1.out‚135150.833984375 disk.sda1.in‚0$ memory_percent‚98.91443252563477 network.sent‚ 1.104385e+07) disk./.usagePercent‚78.92907153563232 disk.sda2.in‚0" disk.dm-0.out‚135150.833984375 disk.sda3.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 209.890625 disk.drbd1260.in‚0 disk.sda.out‚ 209.890625 disk.sda3.out‚ 209.890625" disk.nvme0n1.in‚16807.94140625" proc.memory.rssMB‚ 2917.5859375 cpu‚25.382674741604724la•52¸- _wandbruntime‚7144  _runtime‚7144c1—ÌŹ¿ _step‚345 % lr-AdamW/pg1‚0.000752981193380206 % lr-AdamW/pg2‚0.000752981193380206 trainer/global_step‚4249 % _timestamp‚1.7885279066675165e+09 _runtime‚7144.887116396'@漀 % _timestamp‚1.7885279066675165e+09 _runtime‚7144.887116396 _step‚345 % lr-AdamW/pg1‚0.000752981193380206 % lr-AdamW/pg2‚0.000752981193380206 trainer/global_step‚4249Ù 2»- _wandbruntime‚7144  _runtime‚7144i×#QÞ'¼ØO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚181102676675584 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8436873222144 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚191494262450688 $ _timestamp‚1.788527906668718e+09 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚182912343595520 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚191493945794048 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608ªË¿½ÝO UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚181102676675584 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8436873222144 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚191494262450688 $ _timestamp‚1.788527906668718e+09 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚182912343595520 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚191493945794048 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582Ú$˜g„2¾- _wandbruntime‚7146  _runtime‚7146?² Vê&¿äM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚172705107493376 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚181144178423296 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚172800832076366 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 196062935 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚80606712 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8438837248000 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚80606712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚181143926738432 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 312449024 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚164385929171072 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8414919944654 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 115456221 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚48135920 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 196062935 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚182954027565568 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 115456221 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚664 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 288508928 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚164385912131712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 196036563 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8583884678144 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8583596169216 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚172705089490432 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78764438 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8415148925946 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚172705089490432 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚666 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30629184 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚80606714 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 115429851 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30628520 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚191537623734784 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚182954051505664 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚172801078097018 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 trainer/global_step‚4249 _step‚347 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 196036563 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚48135918 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78764438 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8415148925946 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚172705089490432 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚666 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30629184 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚80606714 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 115429851 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30628520 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚191537623734784 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚182954051505664 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚172801078097018 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 trainer/global_step‚4249 _step‚347 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 196036563 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚48135918 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚172705107493376 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚181144178423296 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚172800832076366 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 196062935 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚80606712 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8438837248000 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚80606712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚181143926738432 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 312449024 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚164385929171072 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8414919944654 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 115456221 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚48135920 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 196062935 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚182954027565568 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 115456221 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚664 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 288508928 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚164385912131712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 196036563 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8583884678144 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8583596169216 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496Û0-7B2Á- _wandbruntime‚7146  _runtime‚7146iš–XÂÒ 0loss/FBt2/assignment_loss‚0.3698847261806278 0loss/SRqqt2/detection_loss‚3.104559365998615 .loss/FBt4/detection_loss‚0.984246671086911 2loss/FBt3/assignment_loss‚0.023154986707959326 /loss/FRt1/detection_loss‚2.6206865260998407 0loss/FRt3/assignment_loss‚1.7848601096966228 2loss/SRqqt1/assignment_loss‚1.3249225234443491 0loss/FBt4/assignment_loss‚0.2141472343241076 .loss/FBt1/detection_loss‚2.684352123737335 .loss/FBt2/detection_loss‚2.048753095300574 %loss/total_loss‚44.53325617353405 /loss/FRt4/assignment_loss‚2.901153046554989 /loss/FRt3/detection_loss‚1.8482550492540735 /loss/FRt4/detection_loss‚1.9418347964477207 1loss/SRqqt3/detection_loss‚1.0496226388754115 1loss/SRqqt4/detection_loss‚2.4977262804382727 epoch‚3 trainer/global_step‚4249 .loss/FRt2/detection_loss‚2.981484636183708 0loss/FRt2/assignment_loss‚3.4205302961411013 2loss/SRqqt2/assignment_loss‚1.3630106323821978 _step‚348 .loss/FBt3/detection_loss‚3.139362025260925 0loss/FBt1/assignment_loss‚0.4514329428803952 0loss/SRqqt1/detection_loss‚1.594712813875892 _runtime‚7146.537082866 % _timestamp‚1.7885279083177319e+09 0loss/FRt1/assignment_loss‚3.1949400703112283 2loss/SRqqt3/assignment_loss‚1.1998960929362241 1loss/SRqqt4/assignment_loss‚1.789727489414968•ED]Ã× 1loss/SRqqt4/assignment_loss‚1.789727489414968 0loss/FBt2/assignment_loss‚0.3698847261806278 0loss/SRqqt2/detection_loss‚3.104559365998615 .loss/FBt4/detection_loss‚0.984246671086911 2loss/FBt3/assignment_loss‚0.023154986707959326 /loss/FRt1/detection_loss‚2.6206865260998407 0loss/FRt3/assignment_loss‚1.7848601096966228 2loss/SRqqt1/assignment_loss‚1.3249225234443491 0loss/FBt4/assignment_loss‚0.2141472343241076 .loss/FBt1/detection_loss‚2.684352123737335 .loss/FBt2/detection_loss‚2.048753095300574 %loss/total_loss‚44.53325617353405 /loss/FRt4/assignment_loss‚2.901153046554989 /loss/FRt3/detection_loss‚1.8482550492540735 /loss/FRt4/detection_loss‚1.9418347964477207 1loss/SRqqt3/detection_loss‚1.0496226388754115 1loss/SRqqt4/detection_loss‚2.4977262804382727 epoch‚3 trainer/global_step‚4249 .loss/FRt2/detection_loss‚2.981484636183708 0loss/FRt2/assignment_loss‚3.4205302961411013 2loss/SRqqt2/assignment_loss‚1.3630106323821978 _step‚348 .loss/FBt3/detection_loss‚3.139362025260925 0loss/FBt1/assignment_loss‚0.4514329428803952 0loss/SRqqt1/detection_loss‚1.594712813875892 _runtime‚7146.537082866 % _timestamp‚1.7885279083177319e+09 0loss/FRt1/assignment_loss‚3.1949400703112283 2loss/SRqqt3/assignment_loss‚1.1998960929362241ÜNnÁì„Ä‚(:ù ¬ŠëÔ†ãËù network.sent‚ 1.1050442e+07 network.recv‚ 2.269563e+06) disk./.usagePercent‚78.92936799235697) disk.nvme0c0n1.out‚135531.50732421875 disk.dm-1.in‚0$ disk.dm-0.out‚135531.50732421875 disk.sda.in‚0 disk.sda3.in‚0 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0& proc.memory.availableMB‚ 178.328125* proc.memory.percent‚17.807531356811523 proc.cpu.threads‚17 disk.dm-1.out‚ 210.6640625 disk.drbd1260.out‚ 0.98046875$ disk.nvme0c0n1.in‚16807.94140625" disk.nvme0n1.in‚16807.94140625' disk.nvme0n1.out‚135531.50732421875 disk.sda3.out‚ 210.6640625$ memory_percent‚98.91157150268555 cpu‚25.382444109496227$ disk./.usageGB‚1176.104621887207 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.dm-0.in‚16807.94140625 disk.sda.out‚ 210.6640625" proc.memory.rssMB‚ 2917.5859375Ô®QÜ‘Å‚(:† ¬ŠëÔηëù gpu.0.gpu‚22.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚111.526' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.3052 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ØΟ³‚Æ‚(:÷ »ŠëÔú–Óø disk.dm-0.in‚16943.47265625 disk.sda.in‚0 disk.sda3.out‚ 211.140625 network.sent‚ 1.1161478e+07 disk.dm-1.out‚ 211.140625 disk.drbd1260.out‚0.984375$ disk.dm-0.out‚135970.65283203125 disk.sda.out‚ 211.140625$ memory_percent‚98.91159534454346( proc.memory.availableMB‚ 178.32421875" proc.memory.rssMB‚ 2917.5859375 disk.drbd1260.in‚0) disk./.usagePercent‚78.85809626284708 disk.sda1.out‚0 disk.dm-1.in‚0 disk.sda2.out‚0 disk.sda3.in‚0* proc.memory.percent‚17.807531356811523 cpu‚25.381933317365082) disk.nvme0c0n1.out‚135970.65283203125" disk.nvme0n1.in‚16943.47265625 disk.sda2.in‚0 proc.cpu.threads‚17 network.recv‚ 2.281299e+06% disk./.usageGB‚1175.0426216125488$ disk.nvme0c0n1.in‚16943.47265625' disk.nvme0n1.out‚135970.65283203125 disk.sda1.in‚0‹Tw}œÇ‚(:‘ »ŠëÔ×ÃŒù gpu.0.gpu‚29.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚33.0 gpu.0.powerWatts‚136.998' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.399599999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0fÍÜȂ(:‘ ÊŠëԌ̀ö gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚33.0 gpu.0.powerWatts‚142.826' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.565199999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0½î%É‚(:ð ÊŠëÔÊ¡áù disk.sda.in‚0' disk.nvme0n1.out‚136311.55908203125 disk.drbd1260.out‚0.984375$ memory_percent‚98.91159534454346* proc.memory.percent‚17.807531356811523 cpu‚25.381306306452757 network.sent‚ 1.116669e+07) disk./.usagePercent‚78.89178736636816 disk.dm-0.in‚16943.47265625$ disk.dm-0.out‚136311.55908203125) disk.nvme0c0n1.out‚136311.55908203125" disk.nvme0n1.in‚16943.47265625 disk.dm-1.out‚211.4375 disk.drbd1260.in‚0 network.r  wecv‚ 2.284958e+06 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.sda3.out‚211.4375$ disk.nvme0c0n1.in‚16943.47265625 disk.sda1.in‚0 disk.sda1.out‚0% disk./.usageGB‚1175.5446434020996 disk.sda.out‚211.4375 disk.dm-1.in‚0( proc.memory.availableMB‚ 178.32421875" proc.memory.rssMB‚ 2917.5859375 proc.cpu.threads‚17n·}‘Ê‚(:† ÙŠëÔ‹¸Íö gpu.0.gpu‚36.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚33.0 gpu.0.powerWatts‚114.479' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.8958 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0<ò÷†Ë‚(:û ÙŠëÔã—Þ÷ disk.sda2.in‚0 disk.dm-0.in‚16943.47265625 disk.sda.in‚0 disk.sda.out‚ 211.98828125 disk.sda3.in‚0 disk.sda3.out‚ 211.98828125 network.sent‚ 1.1172564e+07# disk./.usageGB‚1175.00830078125) disk.nvme0c0n1.out‚136402.85107421875" disk.nvme0n1.in‚16943.47265625 disk.dm-1.out‚ 211.98828125 disk.drbd1260.out‚ 0.98828125$ memory_percent‚98.91483783721924" proc.memory.rssMB‚ 2917.5859375$ disk.nvme0c0n1.in‚16943.47265625' disk.nvme0n1.out‚136402.85107421875 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.sda2.out‚0$ disk.dm-0.out‚136402.85107421875* proc.memory.percent‚17.807531356811523( disk./.usagePercent‚78.8557929630615( proc.memory.availableMB‚ 177.79296875 cpu‚25.38070456453519 proc.cpu.threads‚17 network.recv‚ 2.289482e+06 disk.dm-1.in‚0T$P82Ì- _wandbruntime‚7202  _runtime‚7202„.‰OÇÍÁ & lr-AdamW/pg1‚0.0007478484813615459 & lr-AdamW/pg2‚0.0007478484813615459 trainer/global_step‚4299 % _timestamp‚1.7885279638545408e+09 _runtime‚7202.073971195 _step‚349yýŽÌÎÆ & lr-AdamW/pg1‚0.0007478484813615459 & lr-AdamW/pg2‚0.0007478484813615459 trainer/global_step‚4299 % _timestamp‚1.7885279638545408e+09 _runtime‚7202.073971195 _step‚349Ý SÉr2Ï- _wandbruntime‚7202  _runtime‚7202†^óæß'ÐÙO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚4299 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚81502189 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30969088 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚81502189 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚193678282629120 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚184996566038016 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚174735799141664 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 116767455 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚174630105303040 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 _step‚350 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 198243298 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚174630123305984 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 296910848 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚183165445903360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8535088928768 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30968406 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚684 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8535322597376 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚166224916684848 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚174630105303040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚81502187 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8511128477260 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 116741111 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚174736045162108 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚48665216 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚682 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚81502187 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚183165194231808 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 198269644 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8510899496176 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 116767455 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚183165445903360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8535322597376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚183165194231808 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8681419680256 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 % _timestamp‚1.7885279638552644e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 320850944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚174630123305984 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚166224899645488 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8535088928768 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 116741111 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚79634306 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚48665218 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 _runtime‚ 7202.07788024ê«°¿ä'ÑÞO JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚183165194231808 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 198269644 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8510899496176 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 116767455 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚183165445903360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8535322597376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚183165194231808 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8681419680256 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 % _timestamp‚1.7885279638552644e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 320850944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚174630123305984 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚166224899645488 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8535088928768 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 116741111 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚79634306 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚48665218 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 _runtime‚ 7202.07788024 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚4299 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚81502189 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30969088 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚81502189 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚193678282629120 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚184996566038016 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚174735799141664 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 116767455 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚174630105303040 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 _step‚350 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 198243298 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚174630123305984 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 296910848 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚183165445903360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8535088928768 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30968406 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚684 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8535322597376 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚166224916684848 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚174630105303040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚81502187 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8511128477260 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 116741111 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚174736045162108 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚48665216 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚682 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚81502187 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚174669409342976 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8537052954624 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 198288265 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚174669409342976 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 116767803 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8537052954624 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 116767803 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚81520462 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30975680 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚174775534008726 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 198314637 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8537286636544 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚81520464 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 198288265 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚174669391340032 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _runtime‚7203.124354638 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚183206444294656 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚79651700 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8537286636544 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚193721635957760 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8683702593024 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 116794173 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _step‚351 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚183206695979520 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 292703232 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 316643328 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 % _timestamp‚1.7885279649040651e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚81520464 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚48676018 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚166262430002912 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚48676020 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 198314637 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚185038250008064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚673 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚671 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79651027 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚193721952601088 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8683409889792 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319õ[ 4HÔéM SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.al·é u§&located‚ 116794173 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚185038250008064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚673 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚671 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79651027 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚193721952601088 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8683409889792 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚185038226067968 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 trainer/global_step‚4299 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚174669391340032 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8513086966454 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚174669409342976 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8537052954624 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 198288265 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚174669409342976 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 116767803 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8537052954624 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 116767803 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚81520462 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30975680 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚174775534008726 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 198314637 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8537286636544 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚81520464 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 198288265 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚174669391340032 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _runtime‚7203.124354638 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚183206444294656 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚79651700 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8537286636544 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚193721635957760 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8683702593024 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 116794173 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _step‚351 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚183206695979520 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 292703232 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 316643328 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 % _timestamp‚1.7885279649040651e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚81520464 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚48676018 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚166262430002912 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚48676020 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 198314637 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚176594407152640 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚82415937 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚49205316 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚684 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚176594407152640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚354 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚82415939 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚80517442 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 118079063 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚82415937 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 314559488 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚82415939 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚686 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚195862031393792 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚185227963459584 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚187080764540416 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 200495000 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8609066517768 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚176594425155584 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚176710255053372 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 118079063 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚185227711788032 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8633538304000 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31312126 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 290619392 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8608837536684 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚168101417516688 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 118105407 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚176710501073816 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 % _timestamp‚1.7885280269270566e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚185227711788032 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚168101434556048Š=¿¹ÂåßO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 % _timestamp‚1.7885280269270566e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚185227711788032 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚168101434556048 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 200521346 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 trainer/global_step‚4349 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚176594407152640 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚82415937 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚49205316 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚684 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 N@DeviceStatsMonitor.on_train_batcUµF‚#h_start/active_bytes.all.current‚ 251671552 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚176594407152640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚354 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚82415939 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚80517442 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 118079063 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚82415937 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 314559488 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚82415939 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚686 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚195862031393792 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚185227963459584 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚187080764540416 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 200495000 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8609066517768 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚176594425155584 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚176710255053372 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 118079063 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚185227711788032 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8633538304000 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31312126 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 290619392 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8608837536684 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚168101417516688 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 118105407 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚176710501073816 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273â–"Ÿ_2æ- _wandbruntime‚7266  _runtime‚7266¾ûîê&çäM C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚185269213535744 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8635502343168 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 200539967 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚80534892 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚49216118 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚176749989920434 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8611025006962 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8783570127360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚185268961850880 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚49216120 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚176633711192576 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚686 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚176633693189632 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8635268661248 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚82434212 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚684 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚168138964913472 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚168138947874112 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚82434212 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 200566339 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 118132125 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 200566339 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8635502343168 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8783287909888 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 118105755 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚82434214 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚195906018637824 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 200539967 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚195905712480256 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚187122424570368 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217472 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 _step‚355 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚82434214 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624/Q¢ï&èéM [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 282217472 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 _step‚355 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚82434214 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚185269213535744 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8635502343168 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 200539967 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚80534892 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚49216118 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚176749989920434 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8611025006962 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8783570127360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚185268961850880 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚49216120 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚176633711192576 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚686 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚176633693189632 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8635268661248 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚82434212 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚684 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚168138964913472 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚168138947874112 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚82434212 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 200566339 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 118132125 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 200566339 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8635502343168 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8783287909888 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 118105755 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚82434214 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚195906018637824 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 200539967 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚195905712480256 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚187122424570368 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309ãsŽä2é- _wandbruntime‚7266  _runtime‚7266¤zª_êÙ 2loss/SRqqt1/assignment_loss‚1.5866976782679558 /loss/FRt4/detection_loss‚1.9921693915359513 0loss/SRqqt3/detection_loss‚0.864225167940315 /loss/FBt2/detection_loss‚2.5727031230926514 trainer/global_step‚4349 /loss/FRt2/detection_loss‚3.0381964842478433 2loss/SRqqt2/assignment_loss‚1.8072958011781015 2loss/SRqqt4/assignment_loss‚1.9128492785727276 1loss/FBt1/assignment_loss‚0.29315855661520296 .loss/FRt1/detection_loss‚2.855983223233904 epoch‚3 _step‚356 0loss/FRt1/assignment_loss‚3.0158347163881576 0loss/FRt3/assignment_loss‚1.8842996490352295 0loss/FBt2/assignment_loss‚0.3136304653852694 .loss/FBt3/detection_loss‚2.451884591579437 2loss/SRqqt3/assignment_loss‚0.9740516050344604 /loss/FRt3/detection_loss‚1.7190038944211077 1loss/SRqqt4/detection_loss‚2.3005633389248565 .loss/FBt1/detection_loss‚2.366775115331014 1loss/FBt4/assignment_loss‚0.17601523573362024 1loss/SRqqt1/detection_loss‚1.9467221826314927 0loss/SRqqt2/detection_loss‚2.481555982943504 /loss/FBt4/detection_loss‚0.8831027458856503 /loss/FRt2/assignment_loss‚3.512451818495086 &loss/total_loss‚44.068246178770806 0loss/FRt4/assignment_loss‚3.1186958421831545 3loss/FBt3/assignment_loss‚0.0003802901141170878 % _timestamp‚1.7885280286656523e+09 _runtime‚7266.8864142585kNtdëÞ 0loss/FRt4/assignment_loss‚3.1186958421831545 3loss/FBt3/assignment_loss‚0.0003802901141170878 % _timestamp‚1.7885280286656523e+09 _runtime‚7266.886414258 2loss/SRqqt1/assignment_loss‚1.5866976782679558 /loss/FRt4/detection_loss‚1.9921693915359513 0loss/SRqqt3/detection_loss‚0.864225167940315 /loss/FBt2/detection_loss‚2.5727031230926514 trainer/global_step‚4349 /loss/FRt2/detection_loss‚3.0381964842478433 2loss/SRqqt2/assignment_loss‚1.8072958011781015 2loss/SRqqt4/assignment_loss‚1.9128492785727276 1loss/FBt1/assignment_loss‚0.29315855661520296 .loss/FRt1/detection_loss‚2.855983223233904 epoch‚3 _step‚356 0loss/FRt1/assignment_loss‚3.0158347163881576 0loss/FRt3/assignment_loss‚1.8842996490352295 0loss/FBt2/assignment_loss‚0.3136304653852694 .loss/FBt3/detection_loss‚2.451884591579437 2loss/SRqqt3/assignment_loss‚0.9740516050344604 /loss/FRt3/detection_loss‚1.7190038944211077 1loss/SRqqt4/detection_loss‚2.3005633389248565 .loss/FBt1/detection_loss‚2.366775115331014 1loss/FBt4/assignment_loss‚0.17601523573362024 1loss/SRqqt1/detection_loss‚1.9467221826314927 0loss/SRqqt2/detection_loss‚2.481555982943504 /loss/FBt4/detection_loss‚0.8831027458856503 /loss/FRt2/assignment_loss‚3.512451818495086 &loss/total_loss‚44.068246178770806ä¿}͉ì‚(:þ ¤‹ëÔЙä÷) disk./.usagePercent‚78.92674519516868 disk.drbd1260.out‚ 0.99609375 disk.sda.in‚0 disk.sda.out‚ 215.55078125 disk.sda3.in‚0 disk.sda3.out‚ 215.55078125" disk.nvme0n1.in‚17067.99609375' disk.nvme0n1.out‚137953.19287109375 network.sent‚ 1.1308636e+07 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 215.55078125 network.recv‚ 2.320129e+06 disk.dm-0.in‚17067.99609375$ disk.dm-0.out‚137953.19287109375 disk.sda1.in‚0 disk.sda1.out‚0" proc.memory.rssMB‚ 2917.5859375* proc.memory.percent‚17.807531356811523 proc.cpu.threads‚17% disk./.usageGB‚1176.0655403137207$ disk.nvme0c0n1.in‚17067.99609375) disk.nvme0c0n1.out‚137953.19287109375$ memory_percent‚98.91369342803955( proc.memory.availableMB‚ 177.98046875 cpu‚25.37868379803893n ­˜›í‚(: ¤‹ëÔ¿ÕÛ÷ gpu.0.gpu‚27.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚98.265' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚19.653000000000002 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0€rqÄ‘î‚(:† ³‹ëÔÊ«çö gpu.0.gpu‚11.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚100.371' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.0742 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ïÒWyï‚(:î ³‹ëÔý¼¨ú disk.sda.out‚ 215.9296875 disk.sda3.in‚0' disk.nvme0n1.out‚138179.11181640625( proc.memory.availableMB‚ 178.22265625" proc.memory.rssMB‚ 2917.5859375* proc.memory.percent‚17.807531356811523 disk.sda.in‚0( disk./.usagePercent‚78.9270623885036% disk./.usageGB‚1176.0702667236328# disk.nvme0c0n1.in‚ 17234.8046875! disk.nvme0n1.in‚ 17234.8046875 disk.dm-1.out‚ 215.9296875 disk.sda3.out‚ 215.9296875) disk.nvme0c0n1.out‚138179.11181640625 disk.sda1.in‚0 disk.dm-1.in‚0 cpu‚25.378334479950013 proc.cpu.threads‚17 network.sent‚ 1.141543e+07 disk.drbd1260.out‚1 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 17234.8046875$ disk.dm-0.out‚138179.11181640625 disk.sda1.out‚0$ memory_percent‚98.91221523284912 network.recv‚ 2.324937e+06 disk.drbd1260.in‚0²];£kð‚(:à ‹ëÔˆ†æ÷$ memory_percent‚98.91221523284912 proc.cpu.threads‚17% disk./.usageGB‚1176.0458488464355 disk.sda2.out‚0$ disk.dm-0.out‚138475.39306640625# disk.nvme0c0n1.in‚ 17234.8046875( proc.memory.availableMB‚ 178.22265625 disk.sda.in‚0 disk.sda.out‚216.25) disk.nvme0c0n1.out‚138476.01806640625' disk.nvme0n1.out‚138476.14306640625 disk.sda1.out‚0 disk.dm-1.in‚0 disk.drbd1260.out‚1" proc.memory.rssMB‚ 2917.5859375 network.recv‚ 2.32842e+06) disk./.usagePercent‚78.92542368427694 disk.sda3.in‚0 disk.sda1.in‚0 disk.dm-1.out‚216.25 disk.drbd1260.in‚0* proc.memory.percent‚17.807531356811523 cpu‚25.378056905979783 network.sent‚ 1.1420639e+07 disk.sda2.in‚0 disk.dm-0.in‚ 17234.8046875 disk.sda3.out‚216.25! disk.nvme0n1.in‚ 17234.8046875l+£¨éñ‚(:… ‹ëÔðËã÷ gpu.0.gpu‚23.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚102.135' gpu.0.enforcedPowerLimitWÏx¨Š§atts‚500.0 gpu.0.powerPercent‚20.427 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÌÂX3ò‚(:„ Ñ‹ëԜ˜ö gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚92.852' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.5704 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0{þûzó‚(:ï Ñ‹ëÔ˜Ú½ù disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.out‚138683.5078125 disk.sda.in‚0 disk.sda3.in‚0" proc.memory.rssMB‚ 2917.5859375* proc.memory.percent‚17.807531356811523 cpu‚25.377885958289966 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.dm-1.out‚ 216.60546875 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.00390625 network.sent‚ 1.142767e+07% disk./.usageGB‚1176.6112518310547% disk.nvme0c0n1.out‚138683.5078125# disk.nvme0n1.out‚138683.5078125 network.recv‚ 2.333878e+06! disk.nvme0n1.in‚ 17234.8046875 disk.sda1.out‚0 disk.dm-0.in‚ 17234.8046875 disk.sda.out‚ 216.60546875 disk.sda3.out‚ 216.60546875$ memory_percent‚98.91364574432373( proc.memory.availableMB‚ 177.98828125) disk./.usagePercent‚78.96336835297943# disk.nvme0c0n1.in‚ 17234.8046875w¤œô‚(:‘ à‹ëÔ‚ïöö gpu.0.gpu‚45.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚140.424' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.084799999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0üCPþwõ‚(:ì à‹ëÔž»¢ø! disk.nvme0n1.in‚ 17234.8046875 disk.dm-1.in‚0& proc.memory.availableMB‚ 177.984375 network.recv‚ 2.351607e+06 disk.sda.out‚ 217.17578125 disk.sda3.in‚0 disk.sda3.out‚ 217.17578125 disk.sda1.in‚0 proc.cpu.threads‚17 disk.sda.in‚0# disk.nvme0c0n1.in‚ 17234.8046875# disk.nvme0n1.out‚139082.3359375 disk.sda1.out‚0 disk.dm-1.out‚ 217.17578125 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 network.sent‚ 1.1440554e+07 disk.dm-0.in‚ 17234.8046875 disk.drbd1260.out‚ 1.00390625$ memory_percent‚98.91366958618164" proc.memory.rssMB‚ 2917.5859375* proc.memory.percent‚17.807531356811523 cpu‚25.3775672341584) disk./.usagePercent‚78.96900231078544% disk./.usageGB‚1176.6952018737793 disk.dm-0.out‚139082.3359375% disk.nvme0c0n1.out‚139082.3359375“˜12ö- _wandbruntime‚7345  _runtime‚73453j~»Ç÷Á & lr-AdamW/pg1‚0.0007374038281580839 & lr-AdamW/pg2‚0.0007374038281580839 trainer/global_step‚4399 % _timestamp‚1.7885281075055108e+09 _runtime‚7345.725122214 _step‚357ªÌÇ¥ÌøÆ _step‚357 & lr-AdamW/pg1‚0.0007374038281580839 & lr-AdamW/pg2‚0.0007374038281580839 trainer/global_step‚4399 % _timestamp‚1.7885281075055108e+09 _runtime‚7345.725122214åv#Š2ù- _wandbruntime‚7345  _runtime‚7345T±7ß'úÙO _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8881120174592 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚83329689 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚169977935387888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚169977952427248 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31661981 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚178684710965080 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚187290229344256 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚49745418 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚178558727005184 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8881406599680 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8706775577192 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 286425088 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚83329687 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 202773048 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 119443359 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚178558709002240 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8731520342016 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 119417015 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚189164963042816 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚189164939102720 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 % _timestamp‚1.7885281075068843e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 119443359 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚49745416 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚682 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 119417015 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚81407397 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8707004558276 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚680 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚187290481015808 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8731754010624 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚178684956985524 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚187290229344256 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚49745418 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚178558727005184 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8881406599680 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8706775577192 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 286425088 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚83329687 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 202773048 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 119443359 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚178558709002240 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8731520342016 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 119417015 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚189164963042816 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚189164939102720 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 % _timestamp‚1.7885281075068843e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 119443359 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚49745416 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚682 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 119417015 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚81407397 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8707004558276 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚680 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚187290481015808 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8731754010624 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚178684956985524 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81424889 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _runtime‚7346.639503691 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚663 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚178724199811490 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8708963047470 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8733718049792 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚81425554 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31669334 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚83347964 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8708734066178 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 trainer/global_step‚4399 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 202818041 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 202791669 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚187331479407104 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 _step‚359 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚198089732433920 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚49756218 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8883109361152 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 % _timestamp‚1.7885281084195051e+09 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 320837632 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚178598013042176 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8733718049792 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31668671 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 296897536 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 202818041 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚83347962 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚170015482784672 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚665 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚198090053271552 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚178724445832142 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚189206623072768 UHDevim`ÙôceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8733484367872 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 119470077 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 119443707 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0Ë¿Vï&þéM ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚178598013042176 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 202791669 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 202818041 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚83347962 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚170015482784672 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚665 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚198090053271552 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚178724445832142 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚189206623072768 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8733484367872 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 119470077 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 119443707 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81424889 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _runtime‚7346.639503691 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚663 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚178724199811490 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8708963047470 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8733718049792 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚81425554 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31669334 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚83347964 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8708734066178 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 trainer/global_step‚4399 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 202818041 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 202791669 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚187331479407104 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 _step‚359 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚198089732433920 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚49756218 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8883109361152 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 % _timestamp‚1.7885281084195051e+09 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 320837632 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚178598013042176 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8733718049792 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31668671 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 296897536 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚84243439 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 318753792 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚82297158 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 294813696 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚180523028854784 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 205024750 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 204998404 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚189352998572032 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8804942598784 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8981245603840 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚171854470298448 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 % _timestamp‚1.7885281856508787e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 120781311 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚50285518 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚189352746900480 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8804713617700 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚191249137605120 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8829969717248 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 120754967 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚180659412897232 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚189352746900480 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚84243437 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32012300 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚171854453259088 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚84243437 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚82297818 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚50285516¾P†ô~‘ßO SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8829736048640 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚84243439 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚180523010851840 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8829969717248 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚200230088395264 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚189352998572032 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 trainer/global_step‚4449 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚180523010851840 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚658 _step‚362 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.currentÆUaÛg!‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚189352746900480 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚84243437 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32012300 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚171854453259088 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚84243437 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚82297818 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚50285516 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 120754967 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚200230407149056 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8980950790144 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 _runtime‚7423.872148639 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚180523028854784 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚180659166876788 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚191249161545216 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8829736048640 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚84243439 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 318753792 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚82297158 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 294813696 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚180523028854784 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 205024750 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 204998404 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚189352998572032 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8804942598784 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8981245603840 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚171854470298448 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 % _timestamp‚1.7885281856508787e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 120781311 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚50285518 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚189352746900480 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8804713617700 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚191249137605120 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8829969717248 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 120754967 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚180659412897232 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8831933756416 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 120808029 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚191290845515264 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8831933756416 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 _step‚363 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚82315613 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚180698901743850 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚171892000655872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚189393996963328 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚180698655723198 _runtime‚7425.421554399 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8831700074496 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 205069743 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚180562296888832 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚200273773020672 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8983233662976 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8806901087978 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚50296320 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚171891983616512 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚643 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚189394248648192 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚189394248648192 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚180562296888832 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 205069743 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 205043371 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7885281872003834e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 205043371 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚200274079178240 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚191290821575168 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚180562314891776 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚180562314891776 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚82314970 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136q$Ý®ï&–éM P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚180562314891776 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚180562314891776 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚82314970 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8831700074496 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚641 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚84261714 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚84261712 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 trainer/global_step‚4449 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 306157568 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8831933756416 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 120808029 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚191290845515264 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8831933756416 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 _step‚363 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚82315613 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚180698901743850 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚171892000655872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚189393996963328 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚180698655723198 _runtime‚7425.421554399 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8831700074496 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 205069743 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚180562296888832 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚200273773020672 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8983233662976 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8806901087978 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚50296320 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚171891983616512 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚643 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚189394248648192 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚189394248648192 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚180562296888832 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 205069743 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 205043371 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7885281872003834e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 205043371 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚200274079178240 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚191290821575168 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0ëíÚÝ2—- _wandbruntime‚7425  _runtime‚74255É6LZ˜Ô /loss/FRt4/detection_loss‚1.9001839021858848 0loss/SRqqt4/detection_loss‚3.291780630747477 /loss/FRt1/detection_loss‚2.9320777463912964 .loss/FRt2/detection_loss‚3.447357365489006 1loss/SRqqt3/detection_loss‚1.1071174879454904 _runtime‚7425.423144186 epoch‚3 0loss/FRt1/assignment_loss‚3.3737431812286376 0loss/FRt3/assignment_loss‚2.1119576879910062 /loss/FRt4/assignment_loss‚2.612661971495702 /loss/FBt1/assignment_loss‚0.314817604382226 /loss/FRt3/detection_loss‚1.7408680605036873 1loss/SRqqt1/detection_loss‚2.1438487956398413 1loss/SRqqt2/detection_loss‚2.8079925954341887 /loss/FBt2/detection_loss‚2.6808844677039554 /loss/FBt3/detection_loss‚2.7986152321100235 .loss/FRt2/assignment_loss‚4.60872050523758 /loss/FBt1/detection_loss‚2.0417997648841455 % _timestamp‚1.7885281872038558e+09 2loss/SRqqt1/assignment_loss‚1.5638580757536387 2loss/SRqqt2/assignment_loss‚1.9950154453516007 1loss/FBt2/assignment_loss‚0.21283407534265766 1loss/SRqqt4/assignment_loss‚1.809046631058057 0loss/FBt3/assignment_loss‚0.2756349357086947 %loss/total_loss‚48.19514666855859 _step‚364 2loss/SRqqt3/assignment_loss‚1.1924669796632779 1loss/FBt4/assignment_loss‚0.15607629761403322 /loss/FBt4/detection_loss‚1.0757872286964865 trainer/global_step‚4449Iô‰À_™Ù 2loss/SRqqt3/assignment_loss‚1.1924669796632779 1loss/FBt4/assignment_loss‚0.15607629761403322 /loss/FBt4/detection_loss‚1.0757872286964865 trainer/global_step‚4449 /loss/FRt4/detection_loss‚1.9001839021858848 0loss/SRqqt4/detection_loss‚3.291780630747477 /loss/FRt1/detection_loss‚2.9320777463912964 .loss/FRt2/detection_loss‚3.447357365489006 1loss/SRqqt3/detection_loss‚1.1071174879454904 _runtime‚7425.423144186 epoch‚3 0loss/FRt1/assignment_loss‚3.3737431812286376 0loss/FRt3/assignment_loss‚2.1119576879910062 /loss/FRt4/assignment_loss‚2.612661971495702 /loss/FBt1/assignment_loss‚0.314817604382226 /loss/FRt3/detection_loss‚1.7408680605036873 1loss/SRqqt1/detection_loss‚2.1438487956398413 1loss/SRqqt2/detection_loss‚2.8079925954341887 /loss/FBt2/detection_loss‚2.6808844677039554 /loss/FBt3/detection_loss‚2.7986152321100235 .loss/FRt2/assignment_loss‚4.60872050523758 /loss/FBt1/detection_loss‚2.0417997648841455 % _timestamp‚1.7885281872038558e+09 2loss/SRqqt1/assignment_loss‚1.5638580757536387 2loss/SRqqt2/assignment_loss‚1.9950154453516007 1loss/FBt2/assignment_loss‚0.21283407534265766 1loss/SRqqt4/assignment_loss‚1.809046631058057 0loss/FBt3/assignment_loss‚0.2756349357086947 %loss/total_loss‚48.19514666855859 _step‚364ìjÉØáKš‚(:‘ ÉŒëԈєø gpu.0.gpu‚15.0 gpu.0.memory‚1+ gpu.0.fM žQmemoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚138.385' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.676999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µ ë¦{›‚(:ð ÉŒëÔÑÙ·ú$ memory_percent‚98.91748428344727& proc.memory.availableMB‚ 177.359375" proc.memory.rssMB‚ 2917.5859375* proc.memory.percent‚17.807531356811523# disk.nvme0c0n1.in‚ 17605.6640625! disk.nvme0n1.in‚ 17605.6640625 disk.dm-1.out‚ 220.37890625 disk.sda.out‚ 220.37890625 disk.sda3.in‚0 cpu‚25.37586400627746 disk.sda1.out‚0 disk.dm-1.in‚0 disk.sda2.in‚0 disk.sda.in‚0 disk.sda3.out‚ 220.37890625 proc.cpu.threads‚17 network.recv‚ 2.403906e+06) disk./.usagePercent‚79.09479852493784% disk./.usageGB‚1178.5696563720703$ disk.nvme0n1.out‚141295.43359375 disk.sda1.in‚0 disk.drbd1260.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 17605.6640625 network.sent‚ 1.1683611e+07& disk.nvme0c0n1.out‚141295.43359375 disk.drbd1260.out‚ 1.01953125! disk.dm-0.out‚141295.43359375dPVaqœ‚(:æ ØŒëÔÖ¦Þ÷# disk.nvme0c0n1.in‚ 17605.6640625 disk.dm-1.out‚ 220.9140625 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.01953125* proc.memory.percent‚17.807531356811523 network.recv‚ 2.42364e+06) disk./.usagePercent‚79.09480671717894 disk.sda2.out‚0 disk.sda3.out‚ 220.9140625 disk.sda1.in‚0 cpu‚25.375598156368184 network.sent‚ 1.1789173e+07# disk.nvme0c0n1.out‚ 141546.03125! disk.nvme0n1.out‚ 141546.03125 disk.sda1.out‚0( proc.memory.availableMB‚ 177.33203125 proc.cpu.threads‚17 disk.dm-0.in‚ 17605.6640625 disk.dm-0.out‚ 141546.03125 disk.sda.in‚0 disk.sda.out‚ 220.9140625! disk.nvme0n1.in‚ 17605.6640625 disk.dm-1.in‚0$ memory_percent‚98.91765117645264" proc.memory.rssMB‚ 2917.5859375% disk./.usageGB‚1178.5697784423828 disk.sda2.in‚0 disk.sda3.in‚0‰ã(œ‚(:‘ ØŒëÔ³‚æ÷ gpu.0.gpu‚11.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚140.993' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.198600000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 *:‘ž‚(:† çŒëÔû¯Öõ gpu.0.gpu‚32.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚118.749' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.7498 gpu.0.smClock‚1305 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 é5´…Ÿ‚(:ú çŒëÔÁŒ”ø% disk./.usageGB‚1178.5697784423828 disk.sda.in‚0 disk.sda3.in‚0) disk.nvme0c0n1.out‚141887.21240234375! disk.nvme0n1.in‚ 17605.6640625$ memory_percent‚98.91765117645264# disk.nvme0c0n1.in‚ 17605.6640625 disk.sda1.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 221.48046875 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.sda2.out‚0( proc.memory.availableMB‚ 177.33203125 disk.dm-0.in‚ 17605.6640625$ disk.dm-0.out‚141887.21240234375 disk.sda.out‚ 221.48046875 disk.sda3.out‚ 221.48046875* proc.memory.percent‚17.807531356811523 cpu‚25.37534475011412 proc.cpu.threads‚17 network.recv‚ 2.427079e+06' disk.nvme0n1.out‚141887.21240234375 disk.sda1.in‚0 disk.drbd1260.out‚ 1.0234375" proc.memory.rssMB‚ 2917.5859375 network.sent‚ 1.1794184e+07) disk./.usagePercent‚79.09480671717894º<þ¥ ‚(:„ öŒëÔؾæ÷ gpu.0.gpu‚5.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.031' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.6062 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0±ç"‡¡‚(:ü öŒëԿµù disk.sda.in‚0 disk.sda3.out‚ 221.90234375" disk.nvme0n1.in‚17814.08203125' disk.nvme0n1.out‚142376.67724609375 disk.sda1.in‚0 disk.dm-1.out‚ 221.90234375 proc.cpu.threads‚17) disk./.usagePercent‚79.09480671717894% disk./.usageGB‚1178.5697784423828) disk.nvme0c0n1.out‚142376.67724609375 disk.drbd1260.in‚0 disk.dm-0.in‚17814.08203125 cpu‚25.37515284409801 disk.sda.out‚ 221.90234375 disk.sda3.in‚0$ disk.nvme0c0n1.in‚17814.08203125 disk.drbd1260.out‚ 1.0234375 disk.sda2.in‚0$ disk.dm-0.out‚142376.67724609375$ memory_percent‚98.91765117645264( proc.memory.availableMB‚ 177.33203125 network.sent‚ 1.179935e+07 disk.sda1.out‚0 disk.dm-1.in‚0 disk.sda2.out‚0" proc.memory.rssMB‚ 2917.5859375* proc.memory.percent‚17.807531356811523 network.recv‚ 2.430669e+06úl¢‚(:„ …ëÔŠÞà÷ gpu.0.gpu‚9.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.168' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.6336 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%³-w£‚(:ö …ëÔ¤Œþù( disk.nvme0c0n1.out‚142627.2958984375" disk.nvme0n1.in‚17814.08203125 disk.sda2.out‚0 disk.dm-0.in‚17814.08203125( proc.memory.availableMB‚ 177.31640625 disk.sda.in‚0$ disk.nvme0c0n1.in‚17814.08203125& disk.nvme0n1.out‚142627.2958984375 disk.sda2.in‚0* proc.memory.percent‚17.807531356811523 network.sent‚ 1.1805213e+07% disk./.usageGB‚1178.5697784423828 disk.sda3.out‚ 222.390625 disk.sda1.in‚0$ memory_percent‚98.91774654388428" proc.memory.rssMB‚ 2917.5859375 cpu‚25.374900126735785 proc.cpu.threads‚17 network.recv‚ 2.434938e+06 disk.sda3.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 222.390625 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.02734375# disk.dm-0.out‚142627.2958984375) disk./.usagePercent‚79.09480671717894 disk.sda.out‚ 222.390625‡©Ñ72¤- _wandbruntime‚7504  _runtime‚7504OÀ|ÕÅ¥¿ _runtime‚7504.117583934 _step‚365 % lr-AdamW/pg1‚0.000726727478505279 % lr-AdamW/pg2‚0.000726727478505279 trainer/global_step‚4499 % _timestamp‚1.7885282658981445e+09Ù¡¨Ê¦Ä trainer/global_step‚4499 % _timestamp‚1.7885282658981445e+09 _runtime‚7504.117583934 _step‚365 % lr-AdamW/pg1‚0.000726727478505279 % lr-AdamW/pg2‚0.000726727478505279íóÞH}2§- _wandbruntime‚7504  _runtime‚7504‡ý‡à'¨ÚO aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚193333336107520 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 207250106 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚683 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚85157189 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚85157187 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚182633868808940 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚173730988169648 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 122092919 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚191415264456704 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚681 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8927951755264 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚191415516128256 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522240 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚191415264456704 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚182633622788496 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462336 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 _step‚366 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚191415264456704 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚182633622788496 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462336 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 _step‚366 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚182633868808940 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚173730988169648 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 122092919 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚191415264456704 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚681 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8927951755264 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚191415516128256 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522240 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 J6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 207321445 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32368609 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚202457755264000 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚85175462 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 207295073 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 122145981 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8929915781120 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚657 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peF)fak‚ 6741059457 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚83205684 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 122119611 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚191456766204416 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 292703232 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8929915781120 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚193375044017664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9083027889664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8904839128486 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 122145981 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚4499 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 $ _timestamp‚1.788528267485159e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚655 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚173768518527072 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 _step‚367 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚182673357655558 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚182526598738432 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚182526598738432 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 122119611 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚83205027 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 207295073 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚182526616741376 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚191456514519552 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 207321445 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚182526616741376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608çv°¾î&¬èM MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚655 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚173768518527072 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 _step‚367 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚182673357655558 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚182526598738432 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚182526598738432 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 122119611 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚83205027 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 207295073 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚182526616741376 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚191456514519552 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 207321445 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚182526616741376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32369264 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 207321445 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32368609 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚202457755264000 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚85175462 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 207295073 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 122145981 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8929915781120 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚657 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚83205684 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 122119611 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚191456766204416 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 292703232 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8929915781120 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚193375044017664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9083027889664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8904839128486 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 122145981 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚4499 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 $ _timestamp‚1.788528267485159e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736ïZà 2­- _wandbruntime‚7505  _runtime‚7505ïÎN¥a®Û _runtime‚7505.706581292 2loss/SRqqt1/assignment_loss‚1.5080610879737397 /loss/FRt3/detection_loss‚1.3152435399083928 1loss/SRqqt2/detection_loss‚2.3568108177610805 0loss/FRt4/assignment_loss‚2.6818681181745325 0loss/SRqqt4/detection_loss‚2.287850157572673 _step‚368 2loss/SRqqt3/assignment_loss‚0.9856871079907499 /loss/FRt4/detection_loss‚1.7789705182584201 1loss/SRqqt3/detection_loss‚1.0101861505176533 % _timestamp‚1.7885282674870148e+09 %loss/total_loss‚42.35975915142602 epoch‚3 1loss/FBt1/assignment_loss‚0.44419274588472035 .loss/FRt2/detection_loss‚2.956234966554949 /loss/FBt2/detection_loss‚2.3003633519013724 /loss/FBt3/detection_loss‚3.1756922528147697 /loss/FRt1/assignment_loss‚2.895159575072202 0loss/FRt2/assignment_loss‚3.2615190629036195 2loss/SRqqt4/assignment_loss‚1.9471612099844675 1loss/FBt2/assignment_loss‚0.20472839370146248 /loss/FBt1/detection_loss‚2.1307212188839912 /loss/FRt3/assignment_loss‚1.559267182919112 trainer/global_step‚4499 1loss/SRqqt1/detection_loss‚2.0829361839138945 /loss/FBt4/detection_loss‚0.6769688949266921 2loss/SRqqt2/assignment_loss‚1.8266033487660545 3loss/FBt3/assignment_loss‚0.0066064535931218415 1loss/FBt4/assignment_loss‚0.17672508245414997 /loss/FRt1/detection_loss‚2.7902017289941963GÒ'ªf¯à /loss/FBt1/detection_loss‚2.1307212188839912 /loss/FRt3/assignment_loss‚1.559267182919112 trainer/global_step‚4499 1loss/SRqqt1/detection_loss‚2.0829361839138945 /loss/FBt4/detection_loss‚0.6769688949266921 2loss/SRqqt2/assignment_loss‚1.8266033487660545 3loss/FBt3/assignment_loss‚0.0066064535931218415 1loss/FBt4/assignment_loss‚0.17672508245414997 /loss/FRt1/detection_loss‚2.7902017289941963 _runtime‚7505.706581292 2loss/SRqqt1/assignment_loss‚1.5080610879737397 /loss/FRt3/detection_loss‚1.3152435399083928 1loss/SRqqt2/detection_loss‚2.3568108177610805 0loss/FRt4/assignment_loss‚2.6818681181745325 0loss/SRqqt4/detection_loss‚2.287850157572673 _step‚368 2loss/SRqqt3/assignment_loss‚0.9856871079907499 /loss/FRt4/detection_loss‚1.7789705182584201 1loss/SRqqt3/detection_loss‚1.0101861505176533 % _timestamp‚1.7885282674870148e+09 %loss/total_loss‚42.35975915142602 epoch‚3 1loss/FBt1/assignment_loss‚0.44419274588472035 .loss/FRt2/detection_loss‚2.956234966554949 /loss/FBt2/detection_loss‚2.3003633519013724 /loss/FBt3/detection_loss‚3.1756922528147697 /loss/FRt1/assignment_loss‚2.895159575072202 0loss/FRt2/assignment_loss‚3.2615190629036195 2loss/SRqqt4/assignment_loss‚1.9471612099844675 1loss/FBt2/assignment_loss‚0.20472839370146248ð­zÙ›°‚(: ”ëÔÁ¬ ø gpu.0.gpu‚25.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.059' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚18.611800000000002 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0zÆm±‚(:ö ”ëÔ¤ «ù proc.cpu.threads‚17 disk.sda.out‚ 222.9921875 disk.sda3.out‚ 222.9921875$ disk.nvme0c0n1.in‚17814.08203125 disk.sda1.in‚0* proc.memory.percent‚17.807531356811523 network.recv‚ 2.43968e+06 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda.in‚0( disk.nvme0c0n1.out‚142984.0205078125 disk.sda1.out‚0 disk.dm-1.in‚0 disk.drbd1260.out‚ 1.02734375 network.sent‚ 1.1811413e+07) disk./.usagePercent‚79.09489580780085& disk.nvme0n1.out‚142984.0205078125 disk.dm-1.out‚ 222.9921875$ memory_percent‚98.91786575317383& proc.memory.availableMB‚ 177.296875" proc.memory.rssMB‚ 2917.5859375% disk./.usageGB‚1178.5711059570312 disk.dm-0.in‚17814.08203125# disk.dm-0.out‚142984.0205078125 disk.sda3.in‚0" disk.nvme0n1.in‚17814.08203125 disk.drbd1260.in‚0 cpu‚25.3747231397435873§ÀLœ²‚(:‘ £ëÔ¡Ìð÷ gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚138.508' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.701600000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0{|‰}³‚(:ò £ëÔ¯Çùù disk.sda2.in‚0$ disk.dm-0.out‚143235.01220703125 disk.sda3.out‚ 223.40625$ disk.nvme0c0n1.in‚17814.08203125' disk.nvme0n1.out‚143235.01220703125 disk.dm-1.out‚ 223.40625 network.recv‚ 2.445339e+06 disk.sda.out‚ 223.40625 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.in‚0) disk.nvme0c0n1.out‚143235.01220703125 network.sent‚ 1.1919482e+07 disk.sda3.in‚0" disk.nvme0n1.in‚17814.08203125 disk.drbd1260.out‚1.03125$ memory_percent‚98.91788959503174( proc.memory.availableMB‚ 177.29296875* proc.memory.percent‚17.807531356811523 disk.sda2.out‚0 disk.dm-0.in‚17814.08203125 disk.sda.in‚0 disk.dm-1.in‚0" proc.memory.rssMB‚ 2917.5859375 cpu‚25.374406087234608 proc.cpu.threads‚17( disk./.usagePercent‚79.0189214758945% disk./.usageGB‚1177.4390335083008~Ö1]´‚(:… ²ëÔî·õö gpu.0.gpu‚4.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚117.539' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.5078 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0RKâò~µ‚(:ó ²ëÔÙáÎø$ memory_percent‚98.91788959503174" proc.memory.rssMB‚ 2917.5859375* proc.memory.percent‚17.807531356811523 cpu‚25.37413685204867 proc.cpu.threads‚17" disk.nvme0c0n1.in‚ 17991.515625) disk.nvme0c0n1.out‚143576.20361328125 disk.nvme0n1.in‚ 17991.515625 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda2.out‚0 disk.sda.out‚ 223.75390625 network.sent‚ 1.1924665e+07 disk.sda1.out‚0 disk.drbd1260.out‚1.03125 disk.sda.in‚0( proc.memory.availableMB‚ 177.29296875) disk./.usagePercent‚79.01891968384176# disk./.usageGB‚1177.43900680542' disk.nvme0n1.out‚143576.20361328125 disk.sda1.in‚0 disk.dm-1.out‚ 223.75390625 disk.dm-0.in‚ 17991.515625$ disk.dm-0.out‚143576.20361328125 disk.sda3.in‚0 network.recv‚ 2.448749e+06 disk.sda2.in‚0 disk.sda3.out‚ 223.75390625e3û¶‚(:… ÁëÔ†Œëö gpu.0.gpu‚34.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚110.705' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.141 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0…7EË|·‚(:ñ ÁëÔ“Ʋù network.sent‚ 1.1930637e+07 disk.sda.in‚0 disk.dm-1.out‚ 224.6640625 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.03515625( proc.memory.availableMB‚ 177.28515625% disk./.usageGB‚1176.9814186096191 disk.sda2.in‚0 disk.sda3.in‚0 disk.nvme0n1.in‚ 17991.515625 disk.sda1.out‚0" proc.memory.rssMB‚ 2917.5859375 network.recv‚ 2.453095e+06 disk.sda2.out‚0 disk.dm-0.in‚ 17991.515625& disk.nvme0n1.out‚143951.2236328125$ memory_percent‚98.91793727874756* proc.memory.percent‚17.807531356811523 cpu‚25.37385411252651 proc.cpu.threads‚17( disk./.usagePercent‚78.9882105560795# disk.dm-0.out‚143951.2236328125 disk.sda.out‚ 224.6640625 disk.sda3.out‚ 224.6640625" disk.nvme0c0n1.in‚ 17991.515625( disk.nvme0c0n1.out‚143951.2236328125 disk.sda1.in‚0 disk.dm-1.in‚0æÕ¹s¸‚(:… ÐëÔ›®•ö gpu.0.gpu‚19.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚93.386' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.6772 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬Ž:}¹‚(:ò ÐëÔõ±¢ø cpu‚25.37358537605803) disk./.usagePercent‚78.98978551443014 disk.sda1.out‚0 disk.dm-0.in‚ 17991.515625# disk.dm-0.out‚144268.3212890625 disk.sda3.in‚0& disk.nvme0n1.out‚144268.3212890625$ memory_percent‚98.91793727874756 proc.cpu.threads‚17 network.sent‚ 1.1937086e+07% disk./.usageGB‚1177.0048866271973 disk.sda1.in‚0 disk.drbd1260.out‚ 1.03515625 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda.in‚0" disk.nvme0c0n1.in‚ 17991.515625 disk.dm-1.in‚0 disk.sda.out‚ 225.0078125 disk.sda3.out‚ 225.0078125( disk.nvme0c0n1.out‚144268.3212890625 disk.nvme0n1.in‚ 17991.515625( proc.memory.availableMB‚ 177.28515625" proc.memory.rssMB‚ 2917.5859375 network.recv‚ 2.457848e+06 disk.dm-1.out‚ 225.0078125 disk.drbd1260.in‚0* proc.memory.percent‚17.807531356811523âQp-2º- _wandbruntime‚7581  _runtime‚7581{Î+úÆ»À & lr-AdamW/pg2‚0.0007213051911921511 trainer/global_step‚4549 $ _timestamp‚1.788528343675276e+09 _runtime‚7581.894706982 _step‚369 & lr-AdamW/pg1‚0.0007213051911921511ÖØ<ó˼Š_step‚369 & lr-AdamW/pg1‚0.0007213051911921511 & lr-AdamW/pg2‚0.0007213051911921511 trainer/global_step‚4549 $ _timestamp‚1.788528343675276e+09 _runtime‚7581.894706982ñ72½- _wandbruntime‚7581  _runtime‚7581e§¿ÑÖ¾ÚO TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚86070939 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚193477782012928 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚184451632553984 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 123430871 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚204598103025664 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9180568415744 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚193477782012928 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚184608324720648 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9000589698716 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9026401130496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32705716 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚184451614551040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 123457215 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚184608078700204 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚175607506040848 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 _runtime‚7581.895914458 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚702 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 305299456 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚204598432265216 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚195417558550016 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32706416 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚184451614551040 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚86070937 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 % _timestamp‚1.7885283436763108e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚184451632553984 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9026167461888 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚84071432 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚51365716 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9180873715200 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 123457215 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚86070937 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ á‹@¾ 6966900736 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚700 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 123430871 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚193478033684480 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚195417534609920 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9026401130496 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚193477782012928 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚184451632553984 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 123430871 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚204598103025664 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9180568415744 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚193477782012928 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚184608324720648 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9000589698716 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9026401130496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32705716 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚184451614551040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 123457215 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚184608078700204 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚175607506040848 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 _runtime‚7581.895914458 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚702 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 305299456 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚204598432265216 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚195417558550016 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32706416 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚184451614551040 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚86070937 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 % _timestamp‚1.7885283436763108e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚184451632553984 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9026167461888 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚84071432 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚51365716 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9180873715200 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 123457215 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚86070937 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚700 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 123430871 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚193478033684480 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚195417534609920 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9026401130496 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚184490900588032 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 trainer/global_step‚4549 % _timestamp‚1.7885283445901175e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 123483933 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 209546775 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚184647567546614 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚175645036398272 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚84089302 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚650 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 123457563 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚184490900588032 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 209573147 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚86089214 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9028131487744 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚195459242520064 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚51376520 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9028365169664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 123483933 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚84089954 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚195459218579968 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9182857807360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PÒ î&ÂèM ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚184490900588032 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 trainer/global_step‚4549 % _timestamp‚1.7885283445901175e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 123483933 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 209546775 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚184647567546614 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚175645036398272 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚84089302 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚650 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 123457563 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚184490900588032 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 209573147 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚86089214 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9028131487744 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚195459242520064 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚51376520 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9028365169664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 123483933 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚84089954 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚195459218579968 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9182857807360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273ón²H2Ã- _wandbruntime‚7582  _runtime‚7582 Vü¥ÄÏ .loss/FRt2/detection_loss‚3.091325198610624 .loss/FBt3/detection_loss‚2.415456791718801 /loss/FRt4/detection_loss‚1.6433881896082312 1loss/SRqqt1/detection_loss‚2.1951444879174233 _runtime‚7582.811196196 $loss/total_loss‚45.2044418580087 % _timestamp‚1.7885283445918598e+09 0loss/FRt4/assignment_loss‚2.6130782730049558 2loss/SRqqt1/assignment_loss‚1.4431965475901962 0loss/FBuÐÕ °t1/assignment_loss‚0.1563762491035858 /loss/FBt2/assignment_loss‚0.290189778902069 0loss/SRqqt2/detection_loss‚2.665943858800111 /loss/FBt1/detection_loss‚3.0612581769625344 .loss/FRt3/detection_loss‚1.477573873034041 .loss/FBt2/detection_loss‚2.649230326925005 1loss/FBt3/assignment_loss‚0.07301559052009704 /loss/FRt2/assignment_loss‚3.349920098980268 1loss/FBt4/assignment_loss‚0.17101803116983993 /loss/FBt4/detection_loss‚0.7334395243314566 trainer/global_step‚4549 /loss/FRt1/assignment_loss‚4.005377741960379 /loss/FRt3/assignment_loss‚1.843484869913051 1loss/SRqqt2/assignment_loss‚1.750655975330759 2loss/SRqqt3/assignment_loss‚0.9373910004288842 /loss/FRt1/detection_loss‚3.1315102095787344 1loss/SRqqt3/detection_loss‚0.9118156089305373 epoch‚3 2loss/SRqqt4/assignment_loss‚1.9344656616449356 0loss/SRqqt4/detection_loss‚2.660185793042183 _step‚372ix}ÃZÅÔ 0loss/FBt1/assignment_loss‚0.1563762491035858 /loss/FBt2/assignment_loss‚0.290189778902069 0loss/SRqqt2/detection_loss‚2.665943858800111 /loss/FBt1/detection_loss‚3.0612581769625344 .loss/FRt3/detection_loss‚1.477573873034041 .loss/FBt2/detection_loss‚2.649230326925005 1loss/FBt3/assignment_loss‚0.07301559052009704 /loss/FRt2/assignment_loss‚3.349920098980268 1loss/FBt4/assignment_loss‚0.17101803116983993 /loss/FBt4/detection_loss‚0.7334395243314566 trainer/global_step‚4549 /loss/FRt1/assignment_loss‚4.005377741960379 /loss/FRt3/assignment_loss‚1.843484869913051 1loss/SRqqt2/assignment_loss‚1.750655975330759 2loss/SRqqt3/assignment_loss‚0.9373910004288842 /loss/FRt1/detection_loss‚3.1315102095787344 1loss/SRqqt3/detection_loss‚0.9118156089305373 epoch‚3 2loss/SRqqt4/assignment_loss‚1.9344656616449356 0loss/SRqqt4/detection_loss‚2.660185793042183 _step‚372 .loss/FRt2/detection_loss‚3.091325198610624 .loss/FBt3/detection_loss‚2.415456791718801 /loss/FRt4/detection_loss‚1.6433881896082312 1loss/SRqqt1/detection_loss‚2.1951444879174233 _runtime‚7582.811196196 $loss/total_loss‚45.2044418580087 % _timestamp‚1.7885283445918598e+09 0loss/FRt4/assignment_loss‚2.6130782730049558 2loss/SRqqt1/assignment_loss‚1.4431965475901962ô"¡ãŽÆ‚(:ƒ ßëÔ輌ù gpu.0.gpu‚21.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚99.31' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.862 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 g·3‚Ç‚(:÷ ßëÔðÃâù disk.sda.in‚0$ memory_percent‚98.91831874847412( proc.memory.availableMB‚ 177.22265625* proc.memory.percent‚17.807531356811523 network.sent‚ 1.1941945e+07% disk./.usageGB‚1177.0048866271973 disk.dm-1.in‚0 disk.sda2.in‚0" proc.memory.rssMB‚ 2917.5859375 disk.sda3.in‚0) disk.nvme0c0n1.out‚144374.97021484375 disk.drbd1260.in‚0 disk.sda2.out‚0 disk.sda.out‚ 225.53515625 cpu‚25.3733651067198 proc.cpu.threads‚17 disk.sda3.out‚ 225.53515625 disk.nvme0n1.in‚ 17991.515625' disk.nvme0n1.out‚144374.97021484375 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 225.53515625 disk.drbd1260.out‚ 1.04296875 disk.dm-0.in‚ 17991.515625 network.recv‚ 2.461358e+06) disk./.usagePercent‚78.98978551443014" disk.nvme0c0n1.in‚ 17991.515625$ disk.dm-0.out‚144374.97021484375ü«ŠÎ‘È‚(:† îëÔš£Ôö gpu.0.gpu‚18.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚33.0 gpu.0.powerWatts‚143.982' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.7964 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0®ã3É‚(:ô îëÔ…µè÷ disk.sda2.out‚0 disk.sda3.in‚0 proc.cpu.threads‚17' disk.nvme0n1.out‚144672.18505859375 disk.dm-0.in‚18120.37890625 disk.sda.in‚0 disk.sda.out‚ 225.90625$ memory_percent‚98.91831874847412( proc.memory.availableMB‚ 177.22265625" proc.memory.rssMB‚ 2917.5859375* proc.memory.percent‚17.807531356811523 network.sent‚ 1.2050258e+07 network.recv‚ 2.467115e+06) disk.nvme0c0n1.out‚144672.18505859375 disk.sda1.out‚0 disk.drbd1260.out‚ 1.04296875 disk.sda2.in‚0$ disk.dm-0.out‚144672.18505859375 disk.sda3.out‚ 225.90625) disk./.usagePercent‚78.98979319465617% disk./.usageGB‚1177.0050010681152$ disk.nvme0c0n1.in‚18120.37890625 disk.sda1.in‚0 disk.dm-1.in‚0 disk.drbd1260.in‚0 cpu‚25.3729082924516" disk.nvme0n1.in‚18120.37890625 disk.dm-1.out‚ 225.90625?#~ Ê‚(:„ ýëÔ–¦¶ö gpu.0.gpu‚30.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚33.0 gpu.0.powerWatts‚141.98' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.396 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0àzË‚(:ï ýëÔ‡Ó‰ù) disk./.usagePercent‚78.94606275567985 disk.sda2.in‚0" disk.nvme0n1.in‚18120.37890625& disk.nvme0n1.out‚145140.5478515625 disk.drbd1260.in‚0 disk.drbd1260.out‚1.046875" proc.memory.rssMB‚ 2917.5859375 cpu‚25.37233341005932 network.sent‚ 1.2056052e+07 network.recv‚ 2.471482e+06$ disk./.usageGB‚1176.353385925293 disk.dm-0.in‚18120.37890625 disk.sda.in‚0 disk.sda.out‚ 226.34375 disk.sda3.in‚0 disk.sda3.out‚ 226.34375 disk.sda2.out‚0# disk.dm-0.out‚145140.5478515625$ disk.nvme0c0n1.in‚18120.37890625 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0$ memory_percent‚98.91836643218994( proc.memory.availableMB‚ 177.21484375( disk.nvme0c0n1.out‚145140.5478515625 disk.dm-1.out‚ 226.34375* proc.memory.percent‚17.807531356811523 proc.cpu.threads‚17’t ‘Ì‚(:† ŒŽëÔ¥´åö gpu.0.gpu‚29.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚33.0 gpu.0.powerWatts‚142.174' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.4348 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=Ö²Ù„Í‚(:ù ŒŽëÔƒïëø disk.sda2.in‚0 disk.sda.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 226.73046875( proc.memory.availableMB‚ 177.20703125 cpu‚25.371738567885956 disk.drbd1260.out‚1.046875 network.recv‚ 2.475161e+06) disk./.usagePercent‚78.94606275567985$ disk.nvme0c0n1.in‚18120.37890625 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda.out‚ 226.73046875$ disk./.usageGB‚1176.353385925293 disk.dm-1.out‚ 226.73046875 disk.dm-0.in‚18120.37890625$ memory_percent‚98.91841411590576 proc.cpu.threads‚17 disk.drbd1260.in‚0 network.sent‚ 1.2061329e+07( disk.nvme0c0n1.out‚145548.0283203125& disk.nvme0n1.out‚145548.0283203125 disk.sda2.out‚0# disk.dm-0.out‚145548.0283203125" proc.memory.rssMB‚ 2917.5859375* proc.memory.percent‚17.807531356811523" disk.nvme0n1.in‚18120.37890625 disk.dm-1.in‚0ëŒXù2Î- _wandbruntime‚7636  _runtime‚7636ùÍ7ÝÃϽ % _timestamp‚1.7885283985073981e+09 _runtime‚7636.726924322 _step‚373 $ lr-AdamW/pg1‚0.00071582833168052 $ lr-AdamW/pg2‚0.00071582833168052 trainer/global_step‚45993ºSÈР_runtime‚7636.726924322 _step‚373 $ lr-AdamW/pg1‚0.00071582833168052 $ lr-AdamW/pg2‚0.00071582833168052 trainer/global_step‚4599 % _timestamp‚1.7885283985073981e+09õsüjÚ2Ñ- _wandbruntime‚7636  _runtime‚7636%yÇDà'ÒÚO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚195540551240704 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9124383168512 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9124616837120 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 211779856 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7885283985084252e+09 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33054179 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚197501733112320 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚177484023912048 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚195540299569152 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚186415934403584 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 124795167 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚186415916400640 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚84959995 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚195540299569152 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9098756720308 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚374 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚186415916400640 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚51905818 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚197501757052416 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚86984689 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 211753510 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33054867 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 314559488 _runtime‚7636.727989159 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚4599 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚186582780632356 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9280676012032 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚690 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚86984687 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚84960685 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9280385392640 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9098527739224½=±æå'ÓßO ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚206782433064448 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 124768823 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚177484023912048 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚195540299569152 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚186415934403584 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 124795167 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚186415916400640 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚84959995 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚195540299569152 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9098756720308 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚374 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚186415916400640 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚51905818 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚197501757052416 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚86984689 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 211753510 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33054867 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 314559488 _runtime‚7636.727989159 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚4599 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚186582780632356 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9280676012032 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚690 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚86984687 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚84960685 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9280385392640 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9098527739224 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚195540551240704 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9124383168512 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9124616837120 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 211779856 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7885283985084252e+09 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33054179 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚197501733112320 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚197543441022464 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚87002962 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9126580876288 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚186622269478974 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚186455202437632 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚84977305 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9126347194368 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚186455202437632 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 318740480 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 124795515 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 211824849 EDéòÏ–eviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 211824849 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚51916618 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 294800384 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9100715209502 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9100486228210 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚195581549632000 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚87002964 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚87002962 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚195581801316864 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 124821885 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚186455220440576 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9126347194368 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚197543417082368 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33061389 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚206825791529984 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚186622023458322 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚704 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚177521554269472 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 124821885 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚186455220440576 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚195581801316864 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚84978009 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚375 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9282669248000 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9126580876288 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33060687 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 211798477 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 211798477 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚51916620 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚206826110270464 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚197543441022464 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚87002962 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9126580876288 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚186622269478974 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚186455202437632 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚84977305 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9126347194368 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚186455202437632 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 318740480 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 124795515 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 211824849 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 211824849 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚51916618 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 294800384 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9100715209502 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9100486228210 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚195581549632000 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚87002964 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚87002962 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚195581801316864 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 124821885 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚186455220440576 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9126347194368 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚197543417082368 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33061389 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚206825791529984 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚186622023458322 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚704 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚177521554269472 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 124821885 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚186455220440576 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚195581801316864 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚84978009 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚375 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9282669248000 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9126580876288 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33060687 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 211798477 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 211798477 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚51916620 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚206826110270464 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 126133119 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚87898437 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚199585931614720 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚87898439 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚188556990523620 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 126106775 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚85845297 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 214005212 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚197602817125376 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚188380218250240 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33399381 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 214031558 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚188380236253184 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚188380236253184 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33400029 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚648 trainer/global_step‚4649 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚52445916 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚208966510469632 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚208966198007296 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚87898439 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚52445918 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9196694760816 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 _runtime‚7698.465878079 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 % _timestamp‚1.7885284602459366e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9222598875136 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚197603068796928 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462336 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 126106775 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9222832543744 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9380266392576 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _step‚378 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚87898437 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 126133119 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YEDeviö‹“ ceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚197603068796928 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚197602817125376 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚199585955554816 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚188557236544064 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522240 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚650 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BŸ¥å'çßO ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9222598875136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 214031558 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚188380236253184 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚188380236253184 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33400029 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚648 trainer/global_step‚4649 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚52445916 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚208966510469632 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚208966198007296 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚87898439 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚52445918 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9196694760816 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 _runtime‚7698.465878079 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 % _timestamp‚1.7885284602459366e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9222598875136 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚197603068796928 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462336 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 126106775 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9222832543744 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9380266392576 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _step‚378 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚87898437 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 126133119 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚197603068796928 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚197602817125376 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚199585955554816 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚188557236544064 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522240 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚650 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 126133119 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚87898437 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚199585931614720 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚87898439 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚188556990523620 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 126106775 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚85845297 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 214005212 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚197602817125376 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚188380218250240 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33399381 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 J6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚188596725390682 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 214076551 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚209009879060992 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚199627615584768 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 % _timestamp‚1.7885284611745753e+09 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310351872 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚52456718 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚188419522290176 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9224562900992 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9198653250010 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚197644318873088 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚87916714 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9224796582912 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚85863418 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚188596479370030 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚656 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 126133467 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9224796582912 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚85862762 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33406044 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚199627639524864 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 214076551 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9224562900992 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 126133467 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚179398072140672 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 214050179 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286411776 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚4649 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚87916712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚52456720 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 126159837 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 214050179 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9382549888000 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚87916712 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 126159837 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚197644318873088 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚379 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 _runtime‚7699.395531525 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚188419504287232 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚188419522290176 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚87916714 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608&¢êªï&êéM TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 310351872 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚52456718 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚188419522290176 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9224562900992 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9198653250010 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚197644318873088 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚87916714 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9224796582912 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚85863418 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚188596479370030 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚656 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 126133467 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9224796582912 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚85862762 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33406044 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚199627639524864 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 214076551 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9224562900992 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 126133467 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚179398072140672 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 214050179 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 286411776 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚4649 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚87916712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚52456720 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 126159837 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 214050179 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9382549888000 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚87916712 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 126159837 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚197644318873088 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚379 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 _runtime‚7699.395531525 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚188419504287232 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚188419522290176 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚87916714 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚188419504287232 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33406698 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚209010189412864 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚188596725390682 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 214076551 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚209009879060992 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚199627615584768 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 % _timestamp‚1.7885284611745753e+09 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104ûý&2ë- _wandbruntime‚7699  _runtime‚7699}dµð¡ìÕ %loss/total_loss‚46.27476529462213 trainer/global_step‚4649 0loss/FRt2/assignment_loss‚3.2464663207530977 0loss/FRt4/assignment_loss‚3.2932010112064227 1loss/FBt1/assignment_loss‚0.21389936757226938 0loss/FBt2/assignment_loss‚0.3372605087642197 /loss/FBt1/detection_loss‚2.5469115121023997 epoch‚3 /loss/FBt3/detection_loss‚2.9543335676193236 1loss/SRqqt1/assignment_loss‚2.014424452332385 2loss/SRqqt3/assignment_loss‚0.8628065520877262 1loss/FBt4/assignment_loss‚0.12748379053689973 .loss/FRt1/detection_loss‚2.838280943132216 /loss/FRt1/assignment_loss‚3.235020499075613 2loss/FBt3/assignment_loss‚0.0„4Y_º25411868654191493 _runtime‚7699.398241594 _step‚380 .loss/FRt2/detection_loss‚2.831407340367635 .loss/FBt2/detection_loss‚2.646341150457209 % _timestamp‚1.7885284611781864e+09 1loss/SRqqt3/detection_loss‚0.9702043976148833 /loss/SRqqt4/detection_loss‚2.43553227186203 1loss/SRqqt1/detection_loss‚2.1133457088109218 1loss/SRqqt2/detection_loss‚2.6184436778227487 0loss/FRt3/assignment_loss‚1.9666182157359546 2loss/SRqqt2/assignment_loss‚1.9667595093448957 1loss/SRqqt4/assignment_loss‚3.114791081232183 /loss/FRt3/detection_loss‚1.6496252058238923 /loss/FRt4/detection_loss‚1.6430854729260318 /loss/FBt4/detection_loss‚0.6231108687869791•~`íÚ /loss/FBt4/detection_loss‚0.6231108687869791 %loss/total_loss‚46.27476529462213 trainer/global_step‚4649 0loss/FRt2/assignment_loss‚3.2464663207530977 0loss/FRt4/assignment_loss‚3.2932010112064227 1loss/FBt1/assignment_loss‚0.21389936757226938 0loss/FBt2/assignment_loss‚0.3372605087642197 /loss/FBt1/detection_loss‚2.5469115121023997 epoch‚3 /loss/FBt3/detection_loss‚2.9543335676193236 1loss/SRqqt1/assignment_loss‚2.014424452332385 2loss/SRqqt3/assignment_loss‚0.8628065520877262 1loss/FBt4/assignment_loss‚0.12748379053689973 .loss/FRt1/detection_loss‚2.838280943132216 /loss/FRt1/assignment_loss‚3.235020499075613 2loss/FBt3/assignment_loss‚0.025411868654191493 _runtime‚7699.398241594 _step‚380 .loss/FRt2/detection_loss‚2.831407340367635 .loss/FBt2/detection_loss‚2.646341150457209 % _timestamp‚1.7885284611781864e+09 1loss/SRqqt3/detection_loss‚0.9702043976148833 /loss/SRqqt4/detection_loss‚2.43553227186203 1loss/SRqqt1/detection_loss‚2.1133457088109218 1loss/SRqqt2/detection_loss‚2.6184436778227487 0loss/FRt3/assignment_loss‚1.9666182157359546 2loss/SRqqt2/assignment_loss‚1.9667595093448957 1loss/SRqqt4/assignment_loss‚3.114791081232183 /loss/FRt3/detection_loss‚1.6496252058238923 /loss/FRt4/detection_loss‚1.6430854729260318ü_ääî‚(:… ׎ëÔà‘†ø gpu.0.gpu‚29.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚93.674' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.7348 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ç8oï‚(:ö ׎ëÔ·Ûú% proc.memory.availableMB‚ 176.65625 cpu‚25.36967959804297 disk.dm-1.out‚ 229.05078125 disk.sda2.out‚0 disk.sda.in‚0" proc.memory.rssMB‚ 2917.5859375 network.sent‚ 1.2191957e+07 network.recv‚ 2.496607e+06( disk.nvme0c0n1.out‚147031.5107421875" disk.nvme0n1.in‚18255.09765625 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 229.05078125$ disk./.usageGB‚1176.332649230957& disk.nvme0n1.out‚147031.5107421875 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.05859375 disk.sda.out‚ 229.05078125# memory_percent‚98.9217758178711* proc.memory.percent‚17.807531356811523 proc.cpu.threads‚17 disk.dm-1.in‚0) disk./.usagePercent‚78.94467109872372$ disk.nvme0c0n1.in‚18255.09765625 disk.sda2.in‚0 disk.dm-0.in‚18255.09765625# disk.dm-0.out‚147031.5107421875lÏê¨ð‚(:„ æŽëÔ¾ˆ¢÷ gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚95.151' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚19.0302 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0_ÆŸ€†ñ‚(:û æŽëÔµÐü÷ proc.cpu.threads‚17) disk./.usagePercent‚78.94839088819623% disk./.usageGB‚1176.3880767822266 disk.sda2.out‚0 disk.sda.in‚0 disk.sda1.out‚0 network.sent‚ 1.2300145e+07# disk.dm-0.out‚147292.8037109375$ disk.nvme0c0n1.in‚18255.09765625( disk.nvme0c0n1.out‚147292.8037109375' proc.memory.availableMB‚ 176.8671875* proc.memory.percent‚17.807531356811523 cpu‚25.369370968290447 network.recv‚ 2.502325e+06 disk.sda2.in‚0 disk.dm-0.in‚18255.09765625 disk.sda3.in‚0 disk.sda1.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 229.43359375 disk.drbd1260.in‚0 disk.sda.out‚ 229.43359375 disk.sda3.out‚ 229.43359375" disk.nvme0n1.in‚18255.09765625& disk.nvme0n1.out‚147292.8037109375 disk.drbd1260.out‚ 1.05859375$ memory_percent‚98.92048835754395" proc.memory.rssMB‚ 2917.5859375!¸ÖÈ‘ò‚(:† õŽëÔ¢çéõ gpu.0.gpu‚10.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚137.497' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.4994 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0RÙró‚(:ç õŽëÔ”ÁÍø network.sent‚ 1.2306008e+07% disk.nvme0n1.out‚147576.263671875 disk.sda1.in‚0 disk.sda2.out‚0 disk.dm-0.in‚18395.04296875$ memory_percent‚98.92354011535645' proc.memory.availableMB‚ 176.3671875" proc.memory.rssMB‚ 2917.5859375 disk.sda.in‚0 disk.sda3.out‚229.8125' disk.nvme0c0n1.out‚147576.263671875 disk.dm-1.in‚0" disk.dm-0.out‚147576.263671875* proc.memory.percent‚17.807531356811523 cpu‚25.369136851974716 disk.sda.out‚229.8125 disk.sda3.in‚0 disk.sda1.out‚0 disk.drbd1260.in‚0 proc.cpu.threads‚17 network.recv‚ 2.506452e+06( disk./.usagePercent‚78.9502164779229% disk./.usageGB‚1176.4152793884277$ disk.nvme0c0n1.in‚18395.04296875" disk.nvme0n1.in‚18395.04296875 disk.dm-1.out‚229.8125 disk.drbd1260.out‚1.0625 disk.sda2.in‚0pQ¾˜‘ô‚(:† „ëÔ°Ÿîõ gpu.0.gpu‚40.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚113.801' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.7602 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Çì{õ‚(:ð „ëÔ‚‘çø" proc.memory.rssMB‚ 2917.5859375 proc.cpu.threads‚17) disk./.usagePercent‚78.95021110176468% disk./.usageGB‚1176.4151992797852' disk.nvme0c0n1.out‚147728.677734375% disk.nvme0n1.out‚147728.677734375 disk.drbd1260.out‚1.0625 disk.dm-0.in‚18395.04296875' proc.memory.availableMB‚ 176.8671875 network.recv‚ 2.511229e+06$ disk.nvme0c0n1.in‚18395.04296875" disk.nvme0n1.in‚18395.04296875 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda2.out‚0" disk.dm-0.out‚147728.677734375$ memory_percent‚98.92048835754395 disk.dm-1.out‚ 230.1953125 network.sent‚ 1.2312309e+07 disk.sda3.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0* proc.memory.percent‚17.807531356811523 cpu‚25.36890662419104 disk.sda3.out‚ 230.1953125 disk.sda.in‚0 disk.sda.out‚ 230.1953125[¡%pœö‚(:‘ “ëÔ”ÑÊö gpu.0.gpu‚41.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚123.175' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.634999999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0kĈ †÷‚(:û “ëÔ™Ê×÷ disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1260.out‚ 1.06640625 disk.sda.out‚ 230.6328125$ memory_percent‚98.92048835754395 network.sent‚ 1.2317434e+07) disk./.usagePercent‚78.95021110176468% disk./.usageGB‚1176.4151992797852 disk.dm-1.in‚0 disk.dm-1.out‚ 230.6328125 disk.drbd1260.in‚0 disk.sda2.in‚0" proc.memory.rssMB‚ 2917.5859375 disk.sda3.out‚ 230.6328125' disk.nvme0n1.out‚147819.94482421875 disk.sda2.out‚0$ disk.dm-0.out‚147819.94482421875 disk.sda.in‚0* proc.memory.percent‚17.807531356811523 cpu‚25.368749969653244 disk.sda3.in‚0" disk.nvme0n1.in‚18395.04296875 disk.dm-0.in‚18395.04296875' proc.memory.availableMB‚ 176.8671875 proc.cpu.threads‚17 network.recv‚ 2.514811e+06$ disk.nvme0c0n1.in‚18395.04296875) disk.nvme0c0n1.out‚147819.94482421875]ÄÄl2ø- _wandbruntime‚7777  _runtime‚7777ÉLamÇùÁ & lr-AdamW/pg1‚0.0007047154726739068 & lr-AdamW/pg2‚0.0007047154726739068 trainer/global_step‚4699 % _timestamp‚1.7885285388854332e+09 _runtime‚7777.105842862 _step‚381$5E³ÌúÆ _step‚381 & lr-AdamW/pg1‚0.0007047154726739068 & lr-AdamW/pg2‚0.0007047154726739068 trainer/global_step‚4699 % _timestamp‚1.7885285388854332e+09 _runtime‚7777.105842862ý)³]&2û- _wandbruntime‚7777  _runtime‚7777Üì£>à'üÚO E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 216256914 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚88812189 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33748402 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚211150167496192 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 284327936 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 127444727 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚88812187 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚199665334681600 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9480321707008 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 127471071 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚190344520099840 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚88812187 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚190531446435328 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚86733746 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 % _timestamp‚1.7885285388887284e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚88812189 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚199665334681600 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚672 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 308268032 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚674 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚52986018 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 _runtime‚7777.109786948 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9320814581760 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚181237059654448 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚201670154057216 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 _step‚382 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚190344520099840 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 127471071 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚190344538102784 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚52986016 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚181237042615088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚674 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚52986018 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 _runtime‚7777.109786948 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9320814581760 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚181237059654448 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚201670154057216 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 _step‚382 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚190344520099840 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 127471071 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚190344538102784 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚52986016 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚181237042615088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚199665334681600 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9480321707008 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 127471071 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚190344520099840 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚88812187 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0þÕL¶2þ- _wandbruntime‚7779  _runtime‚7779˜u\1 ÿäM TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚201711814087168 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 _runtime‚7779.215583312 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚211193849159680 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚199706836429312 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚190383806136832 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33755076 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚190383824139776 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9296591290518 trainer/global_step‚4699 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 127471419 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚181274590011872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9322778607616 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9482321484288 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚199706836429312 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9322778607616 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 JDDeviceStatsMon ‘Â[Õitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 216301881 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 127471419 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚181274572972512 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 % _timestamp‚1.7885285409925215e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚669 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚88830464 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚199706584744448 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚86751227 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33754409 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚190383806136832 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚88830462 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚88830462 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚201711838027264 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 216328253 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚190383824139776 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚86751227 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33754409 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚190383806136832 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚88830462 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚88830462 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚201711838027264 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 216328253 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚190383824139776 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚89725937 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9419030288384 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 trainer/global_step‚4749 % _timestamp‚1.7885286316440935e+09 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚677 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 _step‚386 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚201727852237824 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚89725939 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 218534962 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚192308839952384 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚89725937 UGDeviceStatsMonitor?„Â|.on_train_batch_start/allocation.small_pool.allocated‚ 128809023 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚203754328619520 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚192506148367480 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 _runtime‚7869.865315788 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9579925539840 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚183113577525648 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚183113560486288 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9419263956992 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34095371 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34094694 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9419030288384 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚53526116 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚213334566621696 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚213334254159360 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9580214062080 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚192308821949440 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚192308839952384 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚53526118 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 128809023 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚201727852237824,/å'•ßO cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9580214062080 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚192308821949440 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚192308839952384 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚53526118 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 128809023 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚201727852237824 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚192505902347036 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 7134511104 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚273 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚192308821949440 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 218508616 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚89725939 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462336 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522240 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚201728103909376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚87620810 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚203754352559616 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚89725937 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9419030288384 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 trainer/global_step‚4749 % _timestamp‚1.7885286316440935e+09 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚677 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 _step‚386 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚201727852237824 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚89725939 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 218534962 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚192308839952384 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚89725937 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 128809023 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚203754328619520 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚192506148367480 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 _runtime‚7869.865315788 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 233668608 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9579925539840 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚183113577525648 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚183113560486288 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9419263956992 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34095371 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34094694 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9419030288384 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚53526116 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚213334566621696 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚213334254159360‚f D:2–- _wandbruntime‚7871  _runtime‚7871n\A.ê&—äM ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚192348125989376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚53536920 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 218579955 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚192348107986432 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚53536918 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9394300349734 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚87639363 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚201769102300672 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9420994314240 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9581918150656 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚89744212 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚201769353985536 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚213378245286400 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 218553583 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚192348125989376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚89744212 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚183151090843712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚387 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 trainer/global_step‚4749 _runtime‚7871.598936115 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚87638689 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚203796012589568 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚89744214 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚89744214 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚192545637214098 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34102443 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546176 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9420994314240 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 218579955 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 218553583 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚672 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚213377930740224 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9582208756736 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 128809371 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚183151107883072 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292JÒ`˜éM K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 128809371 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚183151107883072 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚192348125989376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚53536920 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 218579955 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚192348107986432 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚53536918 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9394300349734 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚87639363 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚201769102300672 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9420994314240 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9581918150656 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚89744212 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚201769353985536 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚213378245286400 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 218553583 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚192348125989376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 565025792 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚89744212 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 565025792 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚183151090843712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚387 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 trainer/global_step‚4749 _runtime‚7871.598936115 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚87638689 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚130o´Ü 0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚203796012589568 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚89744214 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚89744214 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚192545637214098 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34102443 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 314546176 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9420994314240 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 218579955 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 218553583 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚672 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚213377930740224 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9582208756736ƒƒ¥É2™- _wandbruntime‚7871  _runtime‚7871ëUšÏ 1loss/SRqqt3/assignment_loss‚1.273736619769975 0loss/FBt2/assignment_loss‚0.1811139259333668 /loss/FRt2/detection_loss‚3.4204569003161263 1loss/SRqqt1/detection_loss‚1.8782583669174548 /loss/FRt1/detection_loss‚2.8775831557089284 0loss/FRt4/assignment_loss‚2.7007536584476255 2loss/SRqqt2/assignment_loss‚2.0262990908490286 1loss/FBt1/assignment_loss‚0.27123886524004776 .loss/FBt3/detection_loss‚2.112008374470931 0loss/FBt3/assignment_loss‚0.1178515190456962 1loss/FBt4/assignment_loss‚0.19207918887311662 /loss/FRt3/detection_loss‚1.7907099833906586 0loss/SRqqt2/detection_loss‚2.970031440258026 0loss/SRqqt3/detection_loss‚1.090301547302968 epoch‚3 _runtime‚7871.600582529 /loss/FRt2/assignment_loss‚4.273041612961713 0loss/SRqqt1/assignment_loss‚1.71313180341277 /loss/FRt4/detection_loss‚1.7441363467149578 _step‚388 0loss/FRt1/assignment_loss‚2.8477262143165833 /loss/FBt1/detection_loss‚2.1008260041475295 /loss/FBt2/detection_loss‚2.5805140525102614 /loss/FBt4/detection_loss‚0.8640444612641248 1loss/SRqqt4/detection_loss‚3.1340720873129997 #loss/total_loss‚46.034549508382 -loss/FRt3/assignment_loss‚2.1461359244674 2loss/SRqqt4/assignment_loss‚1.7284983647497076 trainer/global_step‚4749 % _timestamp‚1.7885286333812702e+09—<ßZ›Ô 1loss/SRqqt3/assignment_loss‚1.273736619769975 0loss/FBt2/assignment_loss‚0.1811139259333668 /loss/FRt2/detection_loss‚3.4204569003161263 1loss/SRqqt1/detection_loss‚1.8782583669174548 /loss/FRt1/detection_loss‚2.8775831557089284 0loss/FRt4/assignment_loss‚2.7007536584476255 2loss/SRqqt2/assignment_loss‚2.0262990908490286 1loss/FBt1/assignment_loss‚0.27123886524004776 .loss/FBt3/detection_loss‚2.112008374470931 0loss/FBt3/assignment_loss‚0.1178515190456962 1loss/FBt4/assignment_loss‚0.19207918887311662 /loss/FRt3/detection_loss‚1.7907099833906586 0loss/SRqqt2/detection_loss‚2.970031440258026 0loss/SRqqt3/detection_loss‚1.090301547302968 epoch‚3 _runtime‚7871.600582529 /loss/FRt2/assignment_loss‚4.273041612961713 0loss/SRqqt1/assignment_loss‚1.71313180341277 /loss/FRt4/detection_loss‚1.7441363467149578 _step‚388 0loss/FRt1/assignment_loss‚2.8477262143165833 /loss/FBt1/detection_loss‚2.1008260041475295 /loss/FBt2/detection_loss‚2.5805140525102614 /loss/FBt4/detection_loss‚0.8640444612641248 1loss/SRqqt4/detection_loss‚3.1340720873129997 #loss/total_loss‚46.034549508382 -loss/FRt3/assignment_loss‚2.1461359244674 2loss/SRqqt4/assignment_loss‚1.7284983647497076 trainer/global_step‚4749 % _timestamp‚1.7885286333812702e+09„?vîï‘œ‚(:† üëÔ øìö gpu.0.gpu‚31.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚106.541' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.3082 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0sÏñ=€‚(:õ üëÔ‚÷Ðú% disk./.usageGB‚1176.9418182373047 disk.sda.out‚ 234.3515625& disk.nvme0n1.out‚149625.8193359375 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.drbd1260.out‚1.078125 disk.sda.in‚0$ disk.nvme0c0n1.in‚18688.14453125 disk.dm-1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0$ memory_percent‚98.92339706420898& proc.memory.availableMB‚ 176.390625 cpu‚25.367851705075964) disk./.usagePercent‚78.98555294186654 disk.sda3.in‚0 disk.sda3.out‚ 234.3515625 disk.dm-1.out‚ 234.3515625 disk.dm-0.in‚18688.14453125( disk.nvme0c0n1.out‚149625.8193359375" disk.nvme0n1.in‚18688.14453125 disk.sda1.in‚0# disk.dm-0.out‚149625.8193359375" proc.memory.rssMB‚ 2917.5859375* proc.memory.percent‚17.807531356811523 proc.cpu.threads‚17 network.sent‚ 1.2473063e+07 network.recv‚ 2.545665e+06Ü`[¦ž‚(:… ‹ëÔ踭ö gpu.0.gpu‚5.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚117.976' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.5952 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0-Š¥G‡Ÿ‚(:ü ‹ëÔà®ù" disk.nvme0n1.in‚18688.14453125 disk.sda1.in‚0* proc.memory.percent‚17.807531356811523 network.sent‚ 1.2568082e+07) disk./.usagePercent‚78.98555294186654 disk.sda2.in‚0$ disk.nvme0c0n1.in‚18688.14453125 disk.dm-1.in‚0 disk.dm-1.out‚ 234.83984375 disk.drbd1260.in‚0& proc.memory.availableMB‚ 176.640625% disk./.usageGB‚1176.9418182373047 disk.sda.out‚ 234.83984375 disk.sda3.out‚ 234.83984375) disk.nvme0c0n1.out‚150078.61669921875' disk.nvme0n1.out‚150078.61669921875 disk.sda3.in‚0 disk.sda1.out‚0 disk.drbd1260.out‚ 1.08203125$ memory_percent‚98.92187118530273" proc.memory.rssMB‚ 2917.5859375 cpu‚25.36770000130915 proc.cpu.threads‚17 network.recv‚ 2.551591e+06 disk.sda2.out‚0 disk.dm-0.in‚18688.14453125$ disk.dm-0.out‚150078.61669921875 disk.sda.in‚0 Ý!› ‚(: šëÔÜ–öö gpu.0.gpu‚24.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚139.08' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.816000000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ꪩ‚¡‚(:÷ šëÔ¬éÁ÷ disk.sda.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 235.359375" disk.nvme0n1.in‚18688.14453125 disk.sda1.out‚0 disk.drbd1260.in‚0% disk./.usageGB‚1176.9418182373047$ disk.dm-0.out‚150195.80810546875' disk.nvme0n1.out‚150195.80810546875 disk.sda1.in‚0 disk.dm-1.out‚ 235.359375$ memory_percent‚98.92339706420898 cpu‚25.367521739857455 network.sent‚ 1.2573392e+07 disk.sda2.out‚0 disk.sda.out‚ 235.359375$ disk.nvme0c0n1.in‚18688.14453125 disk.dm-1.in‚0 disk.drbd1260.out‚ 1.08203125& proc.memory.availableMB‚ 176.390625" proc.memory.rssMB‚ 2917.5859375 network.recv‚ 2.555559e+06 disk.sda2.in‚0 disk.dm-0.in‚18688.14453125) disk.nvme0c0n1.out‚150195.80810546875* proc.memory.percent‚17.807531356811523 proc.cpu.threads‚17) disk./.usagePercent‚78.98555294186654﩯Œ‘¢‚(:† ©ëÔÖ÷°ø gpu.0.gpu‚25.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚136.678' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.3356 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0(l£‚(:á ©ëÔꪶù" proc.memory.rssMB‚ 2917.5859375 network.sent‚ 1.2579361e+07) disk./.usagePercent‚78.98555294186654 disk.dm-1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda.out‚ 235.74609375! disk.nvme0n1.out‚ 150331.65625 cpu‚25.367444636587166 disk.sda1.in‚0 disk.sda1.out‚0# memory_percent‚98.9264965057373' proc.memory.availableMB‚ 175.8828125 proc.cpu.threads‚17 disk.dm-1.out‚ 235.74609375 disk.drbd1260.out‚ 1.0859375 disk.sda3.in‚0 disk.sda3.out‚ 235.74609375! disk.nvme0c0n1.in‚ 18837.59375# disk.nvme0c0n1.out‚ 150331.65625* proc.memory.percent‚17.807531356811523 network.recv‚ 2.559975e+06% disk./.usageGB‚1176.9418182373047 disk.drbd1260.in‚0 disk.dm-0.in‚ 18837.59375 disk.dm-0.out‚ 150331.65625 disk.sda.in‚0 disk.nvme0n1.in‚ 18837.59375?ìeü‘¤‚(:† ¸ëÔÌûð÷ gpu.0.gpu‚24.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚107.771' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.5542 gpu.0.smClock‚1290 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0C`–¿t¥‚(:é ¸ëÔ¥¤¹ú* proc.memory.percent‚17.807531356811523 disk.nvme0c0n1.in‚ 18867.4375$ disk.nvme0n1.out‚150693.41015625 disk.drbd1260.out‚ 1.0859375 disk.sda2.out‚0 disk.sda.in‚0 cpu‚25.367280147623774 proc.cpu.threads‚17 network.recv‚ 2.564538e+06% disk./.usageGB‚1176.9418182373047 disk.sda1.out‚0 disk.dm-1.in‚0 disk.sda2.in‚0 disk.dm-0.in‚ 18867.4375 disk.sda.out‚ 236.09765625 disk.sda3.out‚ 236.09765625 network.sent‚ 1.2585462e+07) disk./.usagePercent‚78.98555294186654 disk.sda1.in‚0 disk.dm-1.out‚ 236.09765625! disk.dm-0.out‚150693.41015625 disk.sda3.in‚0$ memory_percent‚98.92189502716064" proc.memory.rssMB‚ 2917.5859375& disk.nvme0c0n1.out‚150693.41015625 disk.nvme0n1.in‚ 18867.4375 disk.drbd1260.in‚0( proc.memory.availableMB‚ 176.63671875 6O¦‚(:… ÇëÔ•‹éö gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚137.736' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.5472 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Mý"Õ{§‚(:ð ÇëÔûÓð÷ disk.nvme0n1.in‚ 18867.4375 disk.sda1.in‚0 disk.sda1.out‚0$ disk.dm-0.out‚150902.07666015625 cpu‚25.367202004143433 proc.cpu.threads‚17 disk.sda.in‚0 disk.sda3.in‚0 disk.nvme0c0n1.in‚ 18867.4375 disk.sda2.in‚0* proc.memory.percent‚17.807531356811523) disk./.usagePercent‚78.98554551764805$ disk./.usageGB‚1176.941707611084 disk.dm-1.in‚0 disk.dm-1.out‚ 236.47265625 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.08984375 disk.sda2.out‚0 disk.dm-0.in‚ 18867.4375 network.sent‚ 1.2590457e+07 disk.sda.out‚ 236.47265625 disk.sda3.out‚ 236.47265625) disk.nvme0c0n1.out‚150902.07666015625' disk.nvme0n1.out‚150902.07666015625# memory_percent‚98.9229679107666' proc.memory.availableMB‚ 176.4609375" proc.memory.rssMB‚ 2917.5859375 network.recv‚ 2.567969e+06H«Ú2¨- _wandbruntime‚7962  _runtime‚7962¼ »!Ç©Á & lr-AdamW/pg1‚0.0006933981646154511 & lr-AdamW/pg2‚0.0006933981646154511 trainer/global_step‚4799 % _timestamp‚1.7885287239152417e+09 _runtime‚7962.134995779 _step‚389 åÌªÆ % _timestamp‚1.7885287239152417e+09 _runtime‚7962.134995779 _step‚389 & lr-AdamW/pg1‚0.0006933981646154511 & lr-AdamW/pg2‚0.0006933981646154511 trainer/global_step‚4799…z?22«- _wandbruntime‚7962  _runtime‚7962v2ãAà'¬ÚO KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚203790621465600 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 220786664 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚205838551062016 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 310365184 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚205838527121920 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚90639687 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚194480604279188 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚90639689 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚713 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚194273141801984 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚90639687 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚215518289759744 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9490508882340 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 130146975 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 130120631 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9490279901256 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚203790369794048 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9680049062912 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 220760318 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚54066218 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9517479663616 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚88509248 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚194273141801984 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚194273123799040 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 % _timestamp‚1.7885287239165685e+09 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚54066216 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚194480358258744 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚184990095396848 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9517245995008 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9517245995008 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚194273123799040 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 trainer/global_step‚4799 _runtime‚7962.136288075 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 130120631 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 _step‚390 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚90639689 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34443032 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1æòÅÍ ­ßO RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚90639689 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26344 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34443032 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚203790621465600 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 220786664 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚205838551062016 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 310365184 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚205838527121920 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 560288777 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚90639687 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚194480604279188 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚90639689 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚713 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚194273141801984 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚90639687 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚215518289759744 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9490508882340 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 130146975 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 130120631 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3532 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9490279901256 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚203790369794048 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9680049062912 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 220760318 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚54066218 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9517479663616 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚88509248 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚194273141801984 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 N@DeviceStatsMonitor.on_train_batch_staržS t/active_bytes.all.current‚ 251671552 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚194273123799040 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 % _timestamp‚1.7885287239165685e+09 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚54066216 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚194480358258744 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚184990095396848 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9517245995008 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9517245995008 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚194273123799040 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 trainer/global_step‚4799 _runtime‚7962.136288075 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 130120631 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 _step‚390†6­ù]2®- _wandbruntime‚7963  _runtime‚7963J(7çê&¯äM OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚88527393 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚90657964 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 284314624 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚203831871541760 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 220805285 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34450373 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚185027608714912 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9519210020864 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚88526714 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 130173693 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚215561982928384 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚194312427838976 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 220831657 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚194519847105154 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚677 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34449696 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 130173693 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚54077020 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 220831657 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9519443702784 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚215562291183104 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚194520093125806 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚194312409836032 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚194312427838976 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚90657962 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚194312409836032 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 220805285 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 130147323 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚203831619856896 _runtime‚7963.313421671 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 % _timestamp‚1.7885287250762422e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚90657964 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9492238390242 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚679 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 220831657 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚194519847105154 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚677 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34449696 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 130173693 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚54077020 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 220831657 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9519443702784 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚215562291183104 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚194520093125806 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚194312409836032 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚194312427838976 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚90657962 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚273 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚194312409836032 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 220805285 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 130147323 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚203831619856896 _runtime‚7963.313421671 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 % _timestamp‚1.7885287250762422e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚90657964 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9492238390242 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26370 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚679 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚90657964 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 284314624 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚203831871541760 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 6458321920 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 220805285 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34450373 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚185027608714912 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9519210020864 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚88526714 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 130173693 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚215561982928384 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚309 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚194312427838976 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300‡®ÝË~2±- _wandbruntime‚7963  _runtime‚7963‡X²Ò 1loss/SRqqt1/assignment_loss‚2.046485955064947 epoch‚3 /loss/FBt2/detection_loss‚2.4081724812002743 2loss/SRqqt4/assignment_loss‚1.7527174179752667 /loss/FBt4/assignment_loss‚0.189676276336055 /loss/FRt3/detection_loss‚1.7251174276073773 0loss/SRqqt1/detection_loss‚1.867634710263122 /loss/FBt1/detection_loss‚2.4082658165379573 _step‚392 /loss/FRt2/assignment_loss‚3.536698840558529 2loss/SRqqt3/assignment_loss‚1.0103539821234178 .loss/FRt2/detection_loss‚3.188926324248314 /loss/FBt4/detection_loss‚0.8749142412773587 /loss/FRt1/assignment_loss‚3.361829448491335 0loss/FBt2/assignment_loss‚0.3295425922644731 2loss/SRqqt2/assignment_loss‚1.8673483110032976 1loss/FBt3/assignment_loss‚0.13703974362059462 1loss/SRqqt2/detection_loss‚2.7464193049818277 0loss/FRt4/assignment_loss‚2.2168277939160665 0loss/FBt1/assignment_loss‚0.5070456166527516 1loss/SRqqt3/detection_loss‚0.9447785193774978 0loss/FRt3/assignment_loss‚1.6456055744834568 trainer/global_step‚4799 $ _timestamp‚1.788528725104747e+09 _runtime‚7963.328765847 /loss/FRt1/detection_loss‚2.7183092534542084 .loss/FRt4/detection_loss‚2.348345925907294 0loss/SRqqt4/detection_loss‚2.406547745068868 /loss/FBt3/detection_loss‚2.6860831635338918 &loss/total_loss‚44.924686465948184–ÐB2]³× 0loss/FRt3/assignment_loss‚1.6456055744834568 trainer/global_step‚4799 $ _timestamp‚1.788528725104747e+09 _runtime‚7963.328765847 /loss/FRt1/detection_loss‚2.7183092534542084 .loss/FRt4/detection_loss‚2.348345925907294 0loss/SRqqt4/detection_loss‚2.406547745068868 /loss/FBt3/detection_loss‚2.6860831635338918 &loss/total_loss‚44.924686465948184 1loss/SRqqt1/assignment_loss‚2.046485955064947 epoch‚3 /loss/FBt2/detection_loss‚2.4081724812002743 2loss/SRqqt4/assignment_loss‚1.7527174179752667 /loss/FBt4/assignment_loss‚0.189676276336055 /loss/FRt3/detection_loss‚1.7251174276073773 0loss/SRqqt1/detection_loss‚1.867634710263122 /loss/FBt1/detection_loss‚2.4082658165379573 _step‚392 /loss/FRt2/assignment_loss‚3.536698840558529 2loss/SRqqt3/assignment_loss‚1.0103539821234178 .loss/FRt2/detection_loss‚3.188926324248314 /loss/FBt4/detection_loss‚0.8749142412773587 /loss/FRt1/assignment_loss‚3.361829448491335 0loss/FBt2/assignment_loss‚0.3295425922644731 2loss/SRqqt2/assignment_loss‚1.8673483110032976 1loss/FBt3/assignment_loss‚0.13703974362059462 1loss/SRqqt2/detection_loss‚2.7464193049818277 0loss/FRt4/assignment_loss‚2.2168277939160665 0loss/FBt1/assignment_loss‚0.5070456166527516 1loss/SRqqt3/detection_loss‚0.9447785193774978ˆìö ;‘´‚(:† ÖëÔ®¥Ä÷ gpu.0.gpu‚11.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚31.0 gpu.0.powerWatts‚121.157' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.2314 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0çHÅ}µ‚(:ò ÖëÔ÷£¼ù) disk./.usagePercent‚78.98398412769673) disk.nvme0c0n1.out‚151186.11962890625 disk.sda1.in‚0 disk.dm-1.in‚0 disk.sda3.in‚0( proc.memory.availableMB‚ 175.96484375 cpu‚25.367065801225873$ disk./.usageGB‚1176.918441772461' disk.nvme0n1.out‚151186.11962890625 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda.in‚0 disk.sda3.out‚ 236.98046875* proc.memory.percent‚17.807531356811523 proc.cpu.threads‚17 disk.nvme0n1.in‚ 18867.4375 disk.sda1.out‚0 disk.drbd1260.in‚0 disk.dm-0.in‚ 18867.4375$ disk.dm-0.out‚151186.11962890625 disk.sda.out‚ 236.98046875" proc.memory.rssMB‚ 2917.5859375 network.recv‚ 2.570707e+06 disk.nvme0c0n1.in‚ 18867.4375 disk.dm-1.out‚ 236.98046875 disk.drbd1260.out‚ 1.08984375$ memory_percent‚98.92599582672119 network.sent‚ 1.2595445e+07~íx¶‚(:… åëÔ¾¥÷ö gpu.0.gpu‚28.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚32.0 gpu.0.powerWatts‚139.035' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.807 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0vÆÃåt·‚(:é åëÔÓ¶ ø disk.sda1.out‚0 disk.sda3.out‚ 237.4140625 network.sent‚ 1.2703701e+07 disk.drbd1260.out‚1.09375 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 18867.4375 disk.sda3.in‚0( disk.nvme0c0n1.out‚151448.3212890625 disk.drbd1260.in‚0 disk.nvme0n1.in‚ 18867.4375$ memory_percent‚98.92451763153076( proc.memory.availableMB‚ 176.20703125* proc.memory.percent‚17.807531356811523 cpu‚25.366948488132923 disk.sda.in‚0 disk.sda.out‚ 237.4140625& disk.nvme0n1.out‚151448.3212890625 disk.sda1.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 237.4140625" proc.memory.rssMB‚ 2917.5859375 proc.cpu.threads‚17 network.recv‚ 2.577294e+06( disk./.usagePercent‚78.9855578060097% disk./.usageGB‚1176.9418907165527# disk.dm-0.out‚151448.3212890625 disk.nvme0c0n1.in‚ 18867.4375ÙA›¸‚(: ôëÔôÉ—÷ gpu.0.gpu‚22.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚104.83' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚20.965999999999998 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0hòU,~¹‚(:ó ôëÔÉ×ãù disk.sda.out‚ 237.7890625 disk.sda3.in‚0 disk.sda3.out‚ 237.7890625( disk.nvme0c0n1.out‚151608.1298828125 disk.dm-1.in‚0 disk.drbd1260.out‚1.09375 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.out‚0$ memory_percent‚98.92230033874512' proc.memory.availableMB‚ 176.5703125* proc.memory.percent‚17.807531356811523 network.sent‚ 1.2709934e+07 disk.dm-1.out‚ 237.7890625" proc.memory.rssMB‚ 2917.5859375 cpu‚25.36675161586028 network.recv‚ 2.582257e+06( disk./.usagePercent‚78.9855578060097% disk./.usageGB‚1176.9418907165527$ disk.nvme0c0n1.in‚19002.44140625" disk.nvme0n1.in‚19002.44140625& disk.nvme0n1.out‚151608.1298828125 disk.sda1.in‚0 proc.cpu.threads‚17 disk.drbd1260.in‚0 disk.dm-0.in‚19002.44140625# disk.dm-0.out‚151608.1298828125 disk.sda.in‚0‚¦R‘º‚(:† ƒ‘ëÔÌŸØö gpu.0.gpu‚22.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚102.254' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.4508 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0p÷éл‚(:ô ƒ‘ëÔì¾Éú proc.cpu.threads‚17 network.recv‚ 2.585503e+06 disk.sda2.in‚0 disk.dm-0.in‚19002.44140625 disk.sda.out‚ 238.390625& disk.nvme0n1.out‚151801.7373046875 disk.dm-1.out‚ 238.390625" proc.memory.rssMB‚ 2917.5859375 disk.drbd1260.in‚0 disk.sda3.out‚ 238.390625$ disk.nvme0c0n1.in‚19002.44140625( disk.nvme0c0n1.out‚151801.7373046875 disk.sda1.in‚0 disk.dm-1.in‚0 network.sent‚ 1.2714957e+07( disk./.usagePercent‚78.9855578060097# disk.dm-0.out‚151801.7373046875 disk.sda1.out‚0$ memory_percent‚98.92385005950928( proc.memory.availableMB‚ 176.31640625* proc.memory.percent‚17.807531356811523% disk./.usageGB‚1176.9418907165527 disk.drbd1260.out‚ 1.09765625 disk.sda2.out‚0 disk.sda.in‚0 disk.sda3.in‚0" disk.nvme0n1.in‚19002.44140625 cpu‚25.366684353712173Xê@¼‚(:… ’‘ëԿס÷ gpu.0.gpu‚14.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚92.922' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.5844 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™¨:潂(:ô ’‘ëÔ‡¸Çø% disk./.usageGB‚1176.8790550231934 disk.sda2.out‚0" disk.nvme0n1.in‚19002.44140625 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.dm-0.in‚19002.44140625# disk.dm-0.out‚152105.7685546875 disk.sda.in‚0& disk.nvme0n1.out‚152105.7685546875 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚98.92382621765137' proc.memory.availableMB‚ 176.3203125( disk.nvme0c0n1.out‚152105.7685546875) disk./.usagePercent‚78.98134084990569 disk.sda2.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 238.859375 disk.drbd1260.out‚ 1.09765625" proc.memory.rssMB‚ 2917.5859375 proc.cpu.threads‚17 disk.sda.out‚ 238.859375 disk.dm-1.out‚ 238.859375* proc.memory.percent‚17.807531356811523 cpu‚25.36660446579022$ disk.nvme0c0n1.in‚19002.44140625 network.sent‚ 1.2720109e+07 network.recv‚ 2.589057e+06šÌâǾ‚(:„ ¡‘ëÔéñ‘ö gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚92.813' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.5626 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»Â¥q|¿‚(:ñ ¡‘ëÔ©µ¢ø disk.sda3.out‚ 239.20703125 disk.sda2.in‚0 disk.sda.in‚0% disk.nvme0c0n1.out‚152371.3984375( proc.memory.availableMB‚ 163.99609375# proc.memory.rssMB‚ 2930.21484375) proc.memory.percent‚17.88461208343506 cpu‚25.371024902513522 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.1015625 disk.dm-1.in‚0 disk.dm-0.in‚19002.44140625 disk.dm-0.out‚152371.3984375$ disk.nvme0c0n1.in‚19002.44140625$ memory_percent‚98.99904727935791 proc.cpu.threads‚17" disk.nvme0n1.in‚19002.44140625 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda.out‚ 239.20703125 disk.sda3.in‚0 network.sent‚ 1.2725967e+07 network.recv‚ 2.59323e+06) disk./.usagePercent‚78.98134084990569% disk./.usageGB‚1176.8790550231934# disk.nvme0n1.out‚152371.3984375 disk.sda1.out‚0 disk.dm-1.out‚ 239.20703125¬ã.À‚(:ë °‘ëÔÈ×ø÷ disk.sda2.out‚0 disk.dm-1.in‚0 cpu‚25.37669367797563 disk.dm-0.in‚19129.55859375 disk.dm-0.out‚152723.5078125$ disk.nvme0c0n1.in‚19129.55859375" disk.nvme0n1.in‚19129.55859375$ memory_percent‚98.99981021881104# ph×X+aroc.memory.rssMB‚ 2930.21484375 disk.sda2.in‚0 disk.sda.in‚0 disk.sda3.out‚ 239.609375# disk.nvme0n1.out‚152723.5078125 disk.sda1.out‚0 disk.drbd1260.in‚0( proc.memory.availableMB‚ 163.87109375) proc.memory.percent‚17.88461208343506 network.sent‚ 1.2732651e+07 disk.sda.out‚ 239.609375 disk.sda3.in‚0% disk.nvme0c0n1.out‚152723.5078125 disk.sda1.in‚0 disk.dm-1.out‚ 239.609375 disk.drbd1260.out‚ 1.1015625 proc.cpu.threads‚17 network.recv‚ 2.598633e+06) disk./.usagePercent‚78.98135569834268% disk./.usageGB‚1176.8792762756348!òæ‘Á‚(:† °‘ëÔË–ò÷ gpu.0.gpu‚13.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚106.586' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.3172 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ãq&œÂ‚(:‘ ¿‘ëÔά§ö gpu.0.gpu‚25.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚103.479' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚20.695800000000002 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ‡†É„Â(:ù ¿‘ëÔÀ¸Ÿù) proc.memory.percent‚17.88461208343506( disk.nvme0c0n1.out‚152837.5419921875 network.sent‚ 1.273776e+07 disk.sda2.in‚0 disk.dm-0.in‚19129.55859375 disk.sda.in‚0 disk.sda3.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0# disk.dm-0.out‚152837.5419921875 disk.sda3.out‚ 240.00390625 disk.sda1.in‚0 disk.dm-1.out‚ 240.00390625 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.10546875& proc.memory.availableMB‚ 164.140625# proc.memory.rssMB‚ 2930.21484375% disk./.usageGB‚1176.8792762756348 disk.sda2.out‚0$ disk.nvme0c0n1.in‚19129.55859375" disk.nvme0n1.in‚19129.55859375$ memory_percent‚98.99816513061523 cpu‚25.382731100954647 proc.cpu.threads‚17 network.recv‚ 2.602055e+06) disk./.usagePercent‚78.98135569834268 disk.sda.out‚ 240.00390625& disk.nvme0n1.out‚152837.5419921875ÚðÇ°Ä‚(:„ ΑëÔæ­ëö gpu.0.gpu‚2.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚92.813' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.5626 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹+ÏT„Å‚(:ù ΑëÔÞ–ˆù network.recv‚ 2.605745e+06) disk./.usagePercent‚78.98135569834268 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda3.in‚0( proc.memory.availableMB‚ 164.21484375* proc.memory.percent‚17.885208129882812 cpu‚25.389493366939476 network.sent‚ 1.2743019e+07$ disk.nvme0c0n1.in‚19129.55859375( disk.nvme0c0n1.out‚153121.5810546875" disk.nvme0n1.in‚19129.55859375& disk.nvme0n1.out‚153121.5810546875 disk.sda1.out‚0 disk.dm-1.in‚0 disk.dm-0.in‚19129.55859375 disk.dm-1.out‚ 240.47265625 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.10546875# disk.dm-0.out‚153121.5810546875 disk.sda.in‚0 disk.sda.out‚ 240.47265625$ memory_percent‚98.99771213531494 proc.memory.rssMB‚ 2930.3125% disk./.usageGB‚1176.8792762756348 disk.sda2.in‚0 disk.sda3.out‚ 240.47265625 proc.cpu.threads‚17Þ>“7Æ‚(:„ Ý‘ëÔÝ׈÷ gpu.0.gpu‚8.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚93.031' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.6062 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0*ï9ÿxÇ‚(:í Ý‘ëÔŸÇáø disk.drbd1260.in‚0 disk.sda.out‚240.9375 disk.sda3.in‚0 disk.sda1.out‚0 disk.drbd1260.out‚1.109375$ memory_percent‚98.94251823425293' proc.memory.availableMB‚ 173.2578125! proc.memory.rssMB‚ 2921.265625) proc.memory.percent‚17.82999038696289 network.sent‚ 1.2748938e+07 disk.sda2.in‚0$ disk.dm-0.out‚153335.10107421875 disk.sda3.out‚240.9375$ disk.nvme0c0n1.in‚19129.55859375' disk.nvme0n1.out‚153335.10107421875 disk.dm-1.in‚0 disk.sda.in‚0) disk.nvme0c0n1.out‚153335.10107421875" disk.nvme0n1.in‚19129.55859375 cpu‚25.39439114783361 proc.cpu.threads‚17 network.recv‚ 2.610281e+06) disk./.usagePercent‚78.98136312256116% disk./.usageGB‚1176.8793869018555 disk.sda2.out‚0 disk.dm-0.in‚19129.55859375 disk.sda1.in‚0 disk.dm-1.out‚240.9375<*È‚(:‚ ì‘ëԟɈ÷ gpu.0.gpu‚14.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚106.7' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.34 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡(yÉ‚(:î ì‘ëÔ‹¶Øø' disk.nvme0n1.out‚153691.28076171875 disk.dm-1.in‚0 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 19243.8984375 disk.sda1.in‚0 disk.drbd1260.out‚1.109375 disk.sda2.out‚0 disk.dm-0.in‚ 19243.8984375$ disk.dm-0.out‚153691.28076171875 disk.sda.out‚ 241.5546875 disk.sda3.out‚ 241.5546875 network.sent‚ 1.2755308e+07 network.recv‚ 2.614787e+06% disk./.usageGB‚1176.8793869018555 disk.sda.in‚0 disk.sda3.in‚0$ memory_percent‚98.99864196777344$ proc.memory.availableMB‚164.0625* proc.memory.percent‚17.885208129882812 disk.sda1.out‚0 proc.memory.rssMB‚ 2930.3125 cpu‚25.4003410817413 proc.cpu.threads‚17 disk.dm-1.out‚ 241.5546875 disk.drbd1260.in‚0) disk./.usagePercent‚78.98136312256116) disk.nvme0c0n1.out‚153691.28076171875! disk.nvme0n1.in‚ 19243.8984375Üîx2Ê- _wandbruntime‚8124  _runtime‚8124èR&‡G*ËÁT RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 9777072785408 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚582 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 223641729 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚206813333364224 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 175812096 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 9613390155776 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 175812096 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚90014228 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 572522496 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 131551436 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚36 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚385 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 223641729 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 161829376 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 560288777 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚273 trainer/global_step‚4835 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 9613214343680 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3532 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚218645632725504 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚208868559940096 _runtime‚8124.958918769 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚32249 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚206813333364224 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚187631124414144 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 6236507136 XIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 7134511104 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚582 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7885288867380037e+09 UHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚92069582 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚197199925205504 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚582 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 7134511104 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 6966900736 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚90014615 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚55038308 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚309 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚309 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚20713 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚197199925205504 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚20713 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚387 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚309 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚218645818494976 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 9777234614784 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 193815040 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚206813139549184 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 6741059457 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 6561988608 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 131572147 cQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚ 9586134507222 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚92069580 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 9613390155776 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚197217275960726 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚20711 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚32006 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 565025792 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚20711 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚32006 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚92069580 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚55038310 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 BDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚92069580 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚55038310 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 6966900736 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚90014615‰‡FDF2Í- _wandbruntime‚8126  _runtime‚8126ÊÁ&nÎÊR WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚55049990 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚34981869 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 193815040 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚90031861 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 6236507136 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚309 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚273 trainer/global_step‚4835 _step‚394 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 175812096 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚32006 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚197230008882688 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 185769472 _runtime‚8126.821288665 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 131560344 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 9613864015872 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 223645306 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚197243118958338 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚206843679083520 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 6458321920 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚20711 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚218676468339200 VGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 7134511104 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 223666019 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 9613688203776 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3532 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚206843872898560 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 6966900736 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚206843679083520 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚273 QBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 7134511104 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 347418624 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚208898921693696 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 6741059457 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32249 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 9613864015872 XJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 565025792 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚90031473 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚218676654108672 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚197243307818170 F>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚582 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚187656684063776 TEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 7134511104 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20713 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 193815040 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚197230008882688 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 565025792 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚36 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 9586606715034 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 175812096 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚187656701103136 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 223645306 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1319 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚55049992 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚32006 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚206843872898560 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 131581055 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚32249 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚92084962 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚197229990879744ý\"U)ÏÏR YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚206843872898560 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 6966900736 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚206843679083520 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚273 QBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 7134511104 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 347418624 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚208898921693696 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 6741059457 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚32249 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 9613864015872 XJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 565025792 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚90031473 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚218676654108672 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚197243307818170 F>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚582 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚187656684063776 TEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 7134511104 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20713 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 193815040 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚197230008882688 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 565025792 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚36 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 9586606715034 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 175812096 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚187656701103136 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 223645306 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1319 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚55049992 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 323478528 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚32006 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚206843872898560 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 131581055 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚32249 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚92084962 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚197229990879744 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚55049990 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚34981869 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 193815040 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚90031861 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 6236507136 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚309 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚273 trainer/global_step‚4835 _step‚394 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 175812096 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚32006 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚197230008882688 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 185769472 _runtime‚8126.821288665 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 131560344 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 9613864015872 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 223645306 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚197243118958338 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚206843679083520 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 6458321920 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚20711 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚218676468339200 VGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 7134511104 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 223666019 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 9613688203776 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3532Š)^j·œÐ‚(:‘ û‘ëÔðã…÷ gpu.0.gpu‚24.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚104.707' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚20.941399999999998 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0’•Û‚Ñ‚(:÷ û‘ëÔ¨†ý÷ disk.sda1.out‚0 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda.out‚ 241.84765625 disk.sda3.in‚0$ memory_percent‚98.99840354919434 network.recv‚ 2.618176e+06% disk./.usageGB‚1176.8793869018555) disk.nvme0c0n1.out‚153736.46728515625! disk.nvme0n1.in‚ 19243.8984375 disk.dm-1.out‚ 241.84765625 disk.sda2.in‚0 cpu‚25.40605935793385 disk.dm-0.in‚ 19243.8984375' proc.memory.availableMB‚ 164.1015625 proc.memory.rssMB‚ 2930.3125* proc.memory.percent‚17.885208129882812 proc.cpu.threads‚17 disk.drbd1260.in‚0 disk.drbd1260.out‚ 1.11328125 network.sent‚ 1.2760204e+07) disk./.usagePercent‚78.98136312256116 disk.sda1.in‚0 disk.sda2.out‚0$ disk.dm-0.out‚153736.46728515625 disk.sda3.out‚ 241.84765625# disk.nvme0c0n1.in‚ 19243.8984375' disk.nvme0n1.out‚153736.46728515625ÛíW«‘Ò‚(:† Š’ëÔï…¼ö gpu.0.gpu‚23.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚103.316' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.6632 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0töb­|Ó‚(:ñ Š’ëÔ¡ÿÑù disk.sda1.out‚0 disk.dm-1.in‚0 disk.drbd1260.in‚0 disk.sda.in‚0$ memory_percent‚98.99821281433105 network.sent‚ 1.2868162e+07( disk./.usagePercent‚78.9813633785687# disk.nvme0c0n1.in‚ 19243.8984375) disk.nvme0c0n1.out‚153992.45166015625' disk.nvme0n1.out‚153992.45166015625 disk.sda3.out‚ 242.578125' proc.memory.availableMB‚ 164.1328125 proc.cpu.threads‚17 network.recv‚ 2.623638e+06 disk.dm-1.out‚ 242.578125 disk.drbd1260.out‚ 1.11328125 disk.sda2.out‚0 disk.dm-0.in‚ 19243.8984375$ disk.dm-0.out‚153992.45166015625 disk.sda.out‚ 242.578125 disk.sda3.in‚0 disk.sda2.in‚0 proc.memory.rssMB‚ 2930.3125* proc.memory.percent‚17.885208129882812 cpu‚25.412016049666036% disk./.usageGB‚1176.8793907165527! disk.nvme0n1.in‚ 19243.8984375 disk.sda1.in‚0Xû©2Ô- _wandbruntime‚8151  _runtime‚8151ë±ô)45Õñ§ & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 jet/accuracy_4_of_6‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.47045740485191345 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4772337079048157 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.37886902689933777 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.13991770148277283 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9970638155937195 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.20562057197093964 8"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.550000011920929 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 *jet/accuracy_3_of_5‚0.0662570372223854 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.02304515428841114 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5270935297012329 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 particle/accuracy_5_of_7‚1 _step‚395 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.08612486720085144 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9942857027053833 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.02631578966975212 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.38517478108406067 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.13075073063373566 +"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.25 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9661487936973572 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 *jet/accuracy_2_of_3‚0.2319050282239914 ) Purity/1FRt3SRqqt1FBt/FRt_purity‚0.25 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.4399999976158142 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9882584810256958 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 +jet/accuracy_1_of_1‚0.47125139832496643 *jet/accuracy_1_of_5‚0.9239760041236877 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.4000000059604645 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ;"Purity/3FRt0SRqqt*FBt/event_purity‚0.011111111380159855 7 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.1600000262260437 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.3499999940395355 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.7058823704719543 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.457572877407074 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5357142686843872 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.47671452164649963 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.48873066902160645 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9824561476707458 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.11925287544727325 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.375 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.015002034604549408 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.33557501435279846 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 8"Purity/1FRt*SRqqt1FBt/event_purity‚0.089467853307724 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.48245614767074585 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 particle/accuracy_1_of_1‚1 particle/accuracy_1_of_2‚1 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 <"Purity/1FRt1SRqqt2FBt/event_purity‚0.0069444444961845875 ,jet/accuracy_4_of_4‚0.007389721926301718 6 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.218528613448143 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.525721549987793 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.36791184544563293 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5816051959991455 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5281670689582825 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.3179738521575928 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.24806439876556396 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7727272510528564 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 (particle/accuracy‚0.8364335298538208 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3240741193294525 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.2708333432674408 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5416666269302368 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.05894104391336441 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 +particle/sensitivity‚0.2675544321537018 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9259259104728699 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.986812174320221 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9862589240074158 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4911755919456482 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5532509088516235 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.009694112464785576 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2702846825122833 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4336131513118744 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ;"Purity/3FRt*SRqqt1FBt/event_purity‚0.008333333767950535 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.023484455421566963 particle/accuracy_2_of_4‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5272014141082764 particle/accuracy_1_of_5‚1 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.13579364120960236 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.18611003458499908 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.12878789007663727 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9464285969734192 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.45104438066482544 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.637512743473053 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.6507936120033264 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9996549487113953 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9994331002235413 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.07762206345796585 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.4204545319080353 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 )jet/accuracy_2_of_4‚0.324400931596756 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.17171718180179596 jet/accuracy_1_of_6‚1 jet/accuracy_6_of_7‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2321428507566452 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.3499999940395355 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.40740740299224854 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13457714021205902 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5229014754295349 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.21883632242679596 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.4444444477558136 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.3813628554344177 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.48061540722846985 particle/accuracy_2_of_2‚1 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.2222222238779068 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.43090274930000305 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 +particle/specificity‚0.9416708946228027 +jet/accuracy_2_of_6‚0.49358975887298584 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.11666060984134674 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1704692542552948 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.014249579980969429 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.052974969148635864 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.2024410218000412 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9909769296646118 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9899606704711914 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.20456261932849884 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_6‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3510968089103699 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9586924910545349 ,&Purity/2FRt2SRqqt3FBt/event_projÇú®Ä^portion‚0 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.2399691492319107 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9873414635658264 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.9166666865348816 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 <"Purity/3FRt*SRqqt*FBt/event_purity‚0.0023419205099344254 ."Purity/0FRt0SRqqt3FBt/event_purity‚0.03125 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.3202614188194275 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 5 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.95634925365448 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9903439283370972 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.03333333507180214 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5805875658988953 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.10116959363222122 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.5475574731826782 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 trainer/global_step‚4835 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9994331002235413 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 'particle/f_score‚0.3379470109939575 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3704688549041748 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.46407532691955566 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 jet/accuracy_4_of_7‚0 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.4166666865348816 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5594576597213745 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7575758099555969 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1742747575044632 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1490938365459442 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9971650242805481 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.25114941596984863 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.36590707302093506 ."Purity/3FRt0SRqqt1FBt/event_purity‚0.03125 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.45573103427886963 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4747858941555023 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9701149463653564 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2361111044883728 particle/accuracy_1_of_6‚1 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.2708333432674408 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6539809107780457 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 6 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.953703761100769 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1981581747531891 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5169752836227417 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.2380952537059784 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8155437111854553 +jet/accuracy_4_of_5‚0.00358422938734293 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.38395407795906067 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9848428964614868 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.4853116273880005 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.4430335462093353 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9790346622467041 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.2083333283662796 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 8"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.349359393119812 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5599886775016785 jet/accuracy_5_of_6‚0 % _timestamp‚1.7885289128063223e+09 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5640533566474915 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9864341616630554 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.8999999761581421 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5687512755393982 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.13408662378787994 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12907879054546356 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 +jet/accuracy_3_of_3‚0.01773827336728573 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.2388889193534851 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.017241379246115685 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5591399669647217 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.1599099189043045 particle/accuracy_3_of_6‚1 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7008928060531616 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.02631578966975212 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.26032373309135437 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9941977262496948 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.2083333283662796 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 epoch‚3 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.14373573660850525 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 jet/accuracy_5_of_7‚0 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1931014358997345 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.4958670139312744 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_6_of_6‚1 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 jet/accuracy_1_of_7‚1 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1450524628162384 particle/accuracy_1_of_4‚1 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.2222222238779068 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5206491947174072 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6474124193191528 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.1427004039287567 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.27599993348121643 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.14270593225955963 6 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.923167884349823 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4549729824066162 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.09579954296350479 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3448459804058075 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.23488986492156982 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.2857142984867096 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9872449040412903 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.1250472366809845 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 particle/accuracy_2_of_7‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.011712750419974327 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.02533692866563797 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2609993815422058 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.44409382343292236 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 *"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.5 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5284721851348877 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.3069068491458893 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.2142857164144516 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9963337182998657 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.43116769194602966 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 _runtime‚8151.030890562 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.29023563861846924 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5786923766136169 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9749352931976318 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.47045740485191345 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9923394322395325 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 +jet/accuracy_3_of_6‚0.09615384787321091 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.17687518894672394 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.01498014759272337 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.007876257412135601 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7239261269569397 particle/accuracy_1_of_3‚1 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.0833333358168602 particle/accuracy_6_of_7‚1 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1704692542552948 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9874708652496338 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.08265931159257889 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5156571269035339 *jet/accuracy_1_of_3‚0.8471975922584534 +"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.25 particle/accuracy_4_of_7‚1 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 7validation_average_jet_accuracy‚0.41833606362342834 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.5625000596046448 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4027778208255768 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.018330127000808716 *jet/accuracy_2_of_5‚0.4243920147418976 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 particle/accuracy_7_of_7‚1 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.75 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6065561175346375 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.13210952281951904 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.21735255420207977 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5328893661499023 particle/accuracy_4_of_4‚1 particle/accuracy_5_of_6‚1 +"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.25 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.43311944603919983 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.71875 7 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.1704980731010437 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 jet/accuracy_6_of_6‚0 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 jet/accuracy_2_of_7‚0 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1269841343164444 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2680346965789795 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ,jet/accuracy_3_of_4‚0.046037524938583374 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5416666269302368 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.15573522448539734 particle/accuracy_2_of_5‚1 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.45736438035964966 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.8976538777351379 particle/accuracy_3_of_4‚1 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 jet/accuracy_7_of_7‚0 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5249620676040649 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9879782199859619 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9266666173934937 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.4608631134033203 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3333333134651184 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.11347518116235733 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.4852393567562103 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5080757141113281 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9648810029029846 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4119349718093872 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.45752137899398804 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5175298452377319 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9246032238006592 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9218750596046448 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6228070259094238 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.5233333110809326 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 <"Purity/2FRt1SRqqt0FBt/event_purity‚0.0019841270986944437 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.6578947305679321 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3816460371017456 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9900521039962769 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9464285969734192 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4861111044883728 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.75 ( Purity/1FRt3SRqqt2FBt/FBt_purity‚0.5 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.16709302365779877 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1477576345205307 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5156084895133972 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9078947305679321 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5055549740791321 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 +"Purity/1FRt*SRqqt3FBt/event_purity‚0.25 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.009429327212274075 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.007751938421279192 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3730654716491699 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 particle/accuracy_5_of_5‚1 particle/accuracy_4_of_6‚1 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9968899488449097 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9987789988517761 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5244252681732178 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2361111044883728 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.16954512894153595 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.19152477383613586 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2746960520744324 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9890351295471191 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 8"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.493827223777771 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.3339498043060303 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_7‚1 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.02073448710143566 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.375347375869751 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.28125 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 +jet/accuracy_2_of_2‚0.09123183786869049 )loss/val_total_loss‚43.92242928312944 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9882672429084778 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.3320748805999756 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.21666666865348816 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.6428571343421936 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9787536263465881 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.550000011920929 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.37534698843955994 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.6666666865348816 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.10550891607999802 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.15905429422855377 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 particle/accuracy_4_of_5‚1 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.15269385278224945 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9704861044883728 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.013153072446584702 particle/accuracy_2_of_3‚1 6 Purity/1FRt3SRqqt*FBt/FBt_purity‚0.970588207244873 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 jet/accuracy_3_of_7‚0 particle/accuracy_3_of_7‚1 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0014430014416575432 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9047619104385376 ( Purity/*FRt3SRqqt2FBt/FBt_purity‚0.5÷«0|.!Öö§ 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.5475574731826782 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 trainer/global_step‚4835 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9994331002235413 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 'particle/f_score‚0.3379470109939575 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3704688549041748 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.46407532691955566 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 jet/accuracy_4_of_7‚0 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.4166666865348816 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5594576597213745 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7575758099555969 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1742747575044632 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1490938365459442 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9971650242805481 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.25114941596984863 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.36590707302093506 ."Purity/3FRt0SRqqt1FBt/event_purity‚0.03125 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.45573103427886963 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4747858941555023 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9701149463653564 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2361111044883728 particle/accuracy_1_of_6‚1 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.2708333432674408 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6539809107780457 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 6 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.953703761100769 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1981581747531891 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5169752836227417 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.2380952537059784 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8155437111854553 +jet/accuracy_4_of_5‚0.00358422938734293 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.38395407795906067 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9848428964614868 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.4853116273880005 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.4430335462093353 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9790346622467041 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.2083333283662796 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 8"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.349359393119812 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5599886775016785 jet/accuracy_5_of_6‚0 % _timestamp‚1.7885289128063223e+09 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5640533566474915 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9864341616630554 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.8999999761581421 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5687512755393982 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.13408662378787994 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12907879054546356 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 +jet/accuracy_3_of_3‚0.01773827336728573 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.2388889193534851 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.017241379246115685 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5591399669647217 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.1599099189043045 particle/accuracy_3_of_6‚1 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7008928060531616 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.02631578966975212 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.26032373309135437 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9941977262496948 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.2083333283662796 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 epoch‚3 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.14373573660850525 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 jet/accuracy_5_of_7‚0 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1931014358997345 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.4958670139312744 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_6_of_6‚1 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 jet/accuracy_1_of_7‚1 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1450524628162384 particle/accuracy_1_of_4‚1 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.2222222238779068 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5206491947174072 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6474124193191528 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.1427004039287567 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.27599993348121643 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.14270593225955963 6 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.923167884349823 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4549729824066162 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.09579954296350479 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3448459804058075 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.23488986492156982 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.2857142984867096 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9872449040412903 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.1250472366809845 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 particle/accuracy_2_of_7‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.011712750419974327 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.02533692866563797 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2609993815422058 8 PÉ áªÏrurity/1FRt0SRqqt1FBt/FRt_purity‚0.44409382343292236 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 *"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.5 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5284721851348877 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.3069068491458893 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.2142857164144516 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9963337182998657 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.43116769194602966 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 _runtime‚8151.030890562 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.29023563861846924 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5786923766136169 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9749352931976318 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.47045740485191345 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9923394322395325 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 +jet/accuracy_3_of_6‚0.09615384787321091 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.17687518894672394 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.01498014759272337 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.007876257412135601 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7239261269569397 particle/accuracy_1_of_3‚1 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.0833333358168602 particle/accuracy_6_of_7‚1 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1704692542552948 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9874708652496338 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.08265931159257889 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5156571269035339 *jet/accuracy_1_of_3‚0.8471975922584534 +"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.25 particle/accuracy_4_of_7‚1 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 7validation_average_jet_accuracy‚0.41833606362342834 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.5625000596046448 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4027778208255768 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.018330127000808716 *jet/accuracy_2_of_5‚0.4243920147418976 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 particle/accuracy_7_of_7‚1 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.75 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6065561175346375 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.13210952281951904 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.21735255420207977 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5328893661499023 particle/accuracy_4_of_4‚1 particle/accuracy_5_of_6‚1 +"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.25 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.43311944603919983 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.71875 7 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.1704980731010437 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 jet/accuracy_6_of_6‚0 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 jet/accuracy_2_of_7‚0 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1269841343164444 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2680346965789795 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ,jet/accuracy_3_of_4‚0.046037524938583374 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5416666269302368 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.15573522448539734 particle/accuracy_2_of_5‚1 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.45736438035964966 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.8976538777351379 particle/accuracy_3_of_4‚1 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 jet/accuracy_7_of_7‚0 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5249620676040649 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9879782199859619 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9266666173934937 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.4608631134033203 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3333333134651184 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.11347518116235733 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.4852393567562103 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5080757141113281 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9648810029029846 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4119349718093872 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.45752137899398804 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5175298452377319 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9246032238006592 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9218750596046448 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6228070259094238 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.5233333110809326 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 <"Purity/2FRt1SRqqt0FBt/event_purity‚0.0019841270986944437 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.6578947305679321 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3816460371017456 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9900521039962769 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9464285969734192 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4861111044883728 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.75 ( Purity/1FRt3SRqqt2FBt/FBt_purity‚0.5 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.16709302365779877 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1477576345205307 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5156084895133972 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9078947305679321 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5055549740791321 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 +"Purity/1FRt*SRqqt3FBt/event_purity‚0.25 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.009429327212274075 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.007751938421279192 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3730654716491699 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 particle/accuracy_5_of_5‚1 particle/accuracy_4_of_6‚1 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9968899488449097 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9987789988517761 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5244252681732178 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2361111044883728 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.16954512894153595 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.19152477383613586 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2746960520744324 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9890351295471191 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 8"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.493827223777771 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.3339498043060303 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_7‚1 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.02073448710143566 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.375347375869751 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.28125 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 +jet/accuracy_2_of_2‚0.09123183786869049 )loss/val_total_loss‚43.92242928312944 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9882672429084778 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.3320748805999756 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.21666666865348816 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.6428571343421936 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9787536263465881 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.550000011920929 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.37534698843955994 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.6666666865348816 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.10550891607999802 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.15905429422855377 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 particle/accuracy_4_of_5‚1 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.15269385278224945 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9704861044883728 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.013153072446584702 particle/accuracy_2_of_3‚1 6 Purity/1FRt3SRqqt*FBt/FBt_purity‚0.970588207244873 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 jet/accuracy_3_of_7‚0 particle/accuracy_3_of_7‚1 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0014430014416575432 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9047619104385376 ( Purity/*FRt3SRqqt2FBt/FBt_purity‚0.5 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 jet/accuracy_4_of_6‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.47045740485191345 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4772337079048157 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.37886902689933777 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.13991770148277283 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9970638155937195 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.20562057197093964 8"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.550000011920929 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 *jet/accuracy_3_of_5‚0.0662570372223854 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.02304515428841114 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5270935297012329 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 particle/accuracy_5_of_7‚1 _step‚395 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.08612486720085144 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9942857027053833 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.02631578966975212 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.38517478108406067 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.13075073063373566 +"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.25 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9661487936973572 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 *jet/accuracy_2_of_3‚0.2319050282239914 ) Purity/1FRt3SRqqt1FBt/FRt_purity‚0.25 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.4399999976158142 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9882584810256958 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 +jet/accuracy_1_of_1‚0.47125139832496643 *jet/accuracy_1_of_5‚0.9239760041236877 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.4000000059604645 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ;"Purity/3FRt0SRqqt*FBt/event_purity‚0.011111111380159855 7 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.1600000262260437 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.3499999940395355 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.7058823704719543 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.457572877407074 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5357142686843872 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.47671452164649963 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.48873066902160645 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9824561476707458 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.11925287544727325 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.375 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.015002034604549408 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.33557501435279846 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 8"Purity/1FRt*SRqqt1FBt/event_purity‚0.089467853307724 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.48245614767074585 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 particle/accuracy_1_of_1‚1 particle/accuracy_1_of_2‚1 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 <"Purity/1FRt1SRqqt2FBt/event_purity‚0.0069444444961845875 ,jet/accuracy_4_of_4‚0.007389721926301718 6 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.218528613448143 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.525721549987793 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.36791184544563293 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5816051959991455 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5281670689582825 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.3179738521575928 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.24806439876556396 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7727272510528564 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 (particle/accuracy‚0.8364335298538208 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3240741193294525 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.2708333432674408 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5416666269302368 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.05894104391336441 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 +particle/sensitivity‚0.2675544321537018 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9259259104728699 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.986812174320221 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9862589240074158 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4911755919456482 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5532509088516235 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.009694112464785576 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2702846825122833 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4336131513118744 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ;"Purity/3FRt*SRqqt1FBt/event_purity‚0.008333333767950535 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.023484455421566963 particle/accuracy_2_of_4‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5272014141082764 particle/accuracy_1_of_5‚1 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.13579364120960236 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.18611003458499908 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.12878789007663727 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9464285969734192 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.45104438066482544 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.637512743473053 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.6507936120033264 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9996549487113953 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9994331002235413 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.07762206345796585 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.4204545319080353 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 )jet/accuracy_2_of_4‚0.324400931596756 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.17171718180179596 jet/accuracy_1_of_6‚1 jet/accuracy_6_of_7‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2321428507566452 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.3499999940395355 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.40740740299224854 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13457714021205902 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5229014754295349 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.21883632242679596 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.4444444477558136 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.3813628554344177 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.48061540722846985 particle/accuracy_2_of_2‚1 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.2222222238779068 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.43090274930000305 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 +particle/specificity‚0.9416708946228027 +jet/accuracy_2_of_6‚0.49358975887298584 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.11666060984134674 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1704692542552948 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.014249579980969429 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.052974969148635864 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.2024410218000412 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9909769296646118 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9899606704711914 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.20456261932849884 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_6‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3510968089103699 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9586924910545349 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.2399691492319107 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9873414635658264 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.9166666865348816 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 <"Purity/3FRt*SRqqt*FBt/event_purity‚0.0023419205099344254 ."Purity/0FRt0SRqqt3FBt/event_purity‚0.03125 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.3202614188194275 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 5 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.95634925365448 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9903439283370972 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.03333333507180214 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5805875658988953 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.10116959363222122 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695‹äÂ6×ׂ(:… ™’ëÔ®…€ø gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚113.954' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.7908 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0N½ÁØ‚(:ö ™’ëÔàÄáø& disk.nvme0n1.out‚154444.7138671875 disk.dm-1.in‚0 disk.drbd1260.in‚0# disk.dm-0.out‚154444.7138671875( proc.memory.availableMB‚ 516.88671875 cpu‚25.41516607034812) disk./.usagePercent‚78.97821371787495 disk.sda1.in‚0 disk.sda.out‚ 243.10546875# memory_percent‚96.8451738357544# proc.memory.rssMB‚ 2924.30859375* proc.memory.percent‚17.848563194274902( disk.nvme0c0n1.out‚154444.7138671875 disk.dm-1.out‚ 243.10546875 disk.drbd1260.out‚ 1.1171875 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 19243.8984375 disk.sda.in‚0 disk.sda3.in‚0% disk./.usageGB‚1176.8324584960938# disk.nvme0c0n1.in‚ 19243.8984375! disk.nvme0n1.in‚ 19243.8984375 disk.sda1.out‚0 disk.sda3.out‚ 243.10546875 proc.cpu.threads‚17 network.sent‚ 1.287397e+07 network.recv‚ 2.627856e+06…ìè¿Ù‚(:ö ¨’ëÔ쌟÷# memory_percent‚96.8449354171753( proc.memory.availableMB‚ 516.92578125# proc.memory.rssMB‚ 2924.30859375% disk./.usageGB‚1176.8324584960938 disk.dm-1.out‚ 243.7109375 disk.drbd1260.in‚0 disk.sda2.in‚0 disk.sda.in‚0 disk.sda3.out‚ 243.7109375 cpu‚25.41496590851727 proc.cpu.threads‚17) disk./.usagePercent‚78.97821371787495( disk.nvme0c0n1.out‚154735.0224609375" disk.nvme0n1.in‚19402.76953125 disk.sda.out‚ 243.7109375* proc.memory.percent‚17.848563194274902 network.sent‚ 1.2999162e+07 network.recv‚ 2.63477e+06$ disk.nvme0c0n1.in‚19402.76953125& disk.nvme0n1.out‚154735.0224609375 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0 disk.drbd1260.out‚ 1.1171875 disk.sda2.out‚0 disk.dm-0.in‚19402.76953125# disk.dm-0.out‚154735.0224609375 disk.sda3.in‚0Ô`œÚ‚(:‘ ¨’ëÔ£ÛŸ÷ gpu.0.gpu‚19.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.950271606445312) gpu.0.memoryAllocatedBytes‚ 7688224768 gpu.0.temp‚30.0 gpu.0.powerWatts‚113.507' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.701400000000003 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚03ƒÐT2Û- _wandbruntime‚8184  _runtime‚8184G|ôÆÜÀ & lr-AdamW/pg1‚0.0006876657837328245 & lr-AdamW/pg2‚0.0006876657837328245 trainer/global_step‚4849 $ _timestamp‚1.788528946082582e+09 _runtime‚8184.303297705 _step‚396†ë ËÝÅ & lr-AdamW/pg1‚0.0006876657837328245 & lr-AdamW/pg2‚0.0006876657837328245 trainer/global_step‚4849 $ _timestamp‚1.788528946082582e+09 _runtime‚8184.303297705 _step‚396Œ¯{‹2Þ- _wandbruntime‚8184  _runtime‚8184ËèÈßÚO JDeviceStatsMonitor.on_ÁØ'train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 132048068 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚55342284 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚207777146675712 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚198131563125760 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 132048068 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9809958895616 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚188476871194864 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚188476854155504 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚656 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚55342282 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚198094838296802 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462336 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 _runtime‚8184.307054418 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚209835540041216 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚209835516101120 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚198095084317246 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚90483905 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚207777146675712 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 132021724 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚198131581128704 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9645565547008 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 224544227 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚92522505 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚90484561 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35142277 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35141623 _step‚397 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9617984141298 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚198131581128704 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 trainer/global_step‚4849 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9618213122382 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 224570573 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9645565547008 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522240 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚198131563125760 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚92522503 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚654 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚92522505 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1319 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9809670373376 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚92522503 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 132021724 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26344 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚582 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 572522496 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚207776895004160 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚207776895004160 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚207776895004160 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚207776895004160 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚582 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 251671552 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 6458321920 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 132048068 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚55342284 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚207777146675712 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚32249 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚309 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26346 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚198131563125760 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 132048068 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚32249 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 6458321920 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9809958895616 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚582 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚188476871194864 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚188476854155504 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 7134511104 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚656 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚273 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 572522496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚55342282 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚198094838296802 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 228981084 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 565025792 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 312462336 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚309 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 251671552 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 233668608 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 7134511104 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚273 _runtime‚8184.307054418 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 565025792 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚209835540041216 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚209835516101120 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚198095084317246 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6561988608 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚90483905 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚207777146675712 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 132021724 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚198131581128704 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9645565547008 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚32006 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6741059457 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 224544227 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚92522505 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6966900736 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚90484561 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26346 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚36 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35142277 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35141623 _step‚397 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9617984141298 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6561988608 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚32006 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚198131581128704 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1300 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 347418624 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6561988608 trainer/global_step‚4849 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 323478528 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9618213122382 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 224570573 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 6236507136 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚309 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 572522496 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9645565547008 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3532 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 288522240 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚198131563125760 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚92522503 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚654 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 246020444 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6966900736 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 109171712 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚582,Çf2á- _wandbruntime‚8186  _runtime‚8186§Çïpé&âãM H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 224615566 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚198170867165696 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9619942630284 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6966900736 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 294800384 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 132074786 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚582 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 228981292 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35149244 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 109171712 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6741059457 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚32249 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚92540778 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 7134511104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 572522496 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚36 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9620171611576 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26372 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 347418624 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚209877224011264 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9647295904256 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 132074786 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 6458321920 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚582 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 233681920 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚209877200071168 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 7134511104 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9811956539392 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 251684864 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9647529586176 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 224589194 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1300 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 246020652 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6561988608 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚207818396751872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 233681920 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚55353086 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚55353084 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚198134573163864 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚207818396751872 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚90501701 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26370 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚198170849162752 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 318740480 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6966900736 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚198170849162752 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9647529586176 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚32006 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚188514401552288 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 572522496 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 224615566 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚219689180550656 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚273 % _timestamp‚1.7885289483353817e+09 _runtime‚ 8186.55945448 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚92540780 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9647295904256 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚582 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 132048416 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582×T£Å ãèM OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 251684864 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚32249 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 132048416 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 560288777 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚582 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 224615566 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚198170867165696 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9619942630284 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1319 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6561988608 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 7134511104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚309 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚273 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 6236507136 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 572522496 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚309 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3532 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P